Skip to content

NOC 20.1

In accordance to our Release Policy <releases-policy> we're proudly present release 20.1.

20.1 release contains 254 bugfixes, optimisations and improvements.

Highlights

Python3 Compatibility

As it was declared in Python3 Compatibility Statement<release-19.4-py3>, 20.1 is the first release which offers full py3 compatibility. Python version can be selected in the Tower interface during deploy.

Following Python versions are used:

  • 2.7 for py2 installations
  • 3.6 for py3 installations

You should stay on py2 if:

  • CLIPS-based config config validation is used. You're urged to move to the ConfDB based validation engine.
  • custom is used. You have to check custom code base for py3 compatibility with tools like 2to3 and futurize. It is advised to check results on separate test installation.
  • pyrules are used. Just in case with custom you have to perform additional compatibility checking and testing is required.
  • You have very personal reason to stay on py2.

All other installation should be deployed in py3 mode.

Please note that Python 3.7+ compatibility cannot be reached just now due to dependencies restriction. Reaching latest stable python version compatibility is still work in progress.

Returning to Python3 Compatibility Statement<release-19.4-py3> -we're starting to remove py2 compatibility code and testing infrastructure in order to clean our codebase. NOC 20.2 will be py3-only release.

Rust infrastructure

NOC beginning to adopt Rust as a viable component of its infrastructure. Rust is blazingly fast and memory-effective language for building reliable software. Common motivation behind the Rust:

  • No language-integrated GC, so it integrates well with Python via PyO3 bindings. Rust can fully replace Cython in the NOC project. Following parts of system can gain great benefit from Rust:
  • ASN.1 BER decoder (SNMP Operation)
  • IP Address manipulation
  • HTTP Client
  • Can be used to build high-performance existing part of the system:
  • ping
  • chwriter
  • syslogcollector
  • trapcollector
  • Can be used to build like new perspective components like:
  • flowcolletor
  • cdrcollector
  • Can be used to write standalone components, like server agents

Though some part of Rust usage held by py3 transition (pyo3 supports Python 3.5+), we're expecting to remove all obstacles in 20.2 release.

We're already rewritten DNS Synchronization component from Go language to Rust.

Physical to Logical Interface Mapping

New mechanism called Collator has been introduced. Collator establishes connection between logical interface (ManagedObject Interface) and hardware slot bound to inventory model. So it is possible to answer the questions:

  • Where particular interface is physically located?
  • Has the particular interface any hardware restrictions?
  • Which logical port corresponds to the physical slot?
  • Which logical ports are served by given hardware module?

Composite Connections

Inventory support for composite connections has been added. Composite connections are physical jacks serving several ports at once. Most known cases are RJ-21/mRJ-21, CENC-36M. Composites are just bundle of pins assigned to several internal connections.

NVD CPE

Inventory support for NVD CPE identifiers has been added. NOC supports CPE v2.2 and v2.3 inventory attributes. NVD CPE identifiers allow to use NOC in security audit process.

Serial Number Checking

Additional inventory model attributes for serial number validity checking has been introduced. Minimal and maximal length of serial number and regular expression pattern can be set.

Binary, broken and non-unique serial numbers are headache on cheap optical modules, so it is good idea to drop obviously crap serial and fallback to NOC-generated ones.

./noc inventory find-serial command

Handy find-serial cli command has been added. It is possible to search for one or more serials from common line and see, where this module is located.

SNMP Display-Hints

SNMP Display-Hints are the first-class citizens. So NOC tries to decode binary OctetString data to the readable textual representation, if defined by MIB or set manually during the function call.

Task Monitor

UI to control discovery jobs has been added.

FM Pools

Now it is possible to separate Managed Object's SA and FM pool binding. Possible cases are:

  • Intersecting address space within single segment
  • Eliminating the need to allocate separate classifier/correlator processes for small SA pools.

NBI getmappings API

NBI API to query object mappings has been introduced. Mappings are relations between identifiers in NOC and in the remote systems. Consider NOC loads Managed Object from external network inventory system (NRI) via ETL process. NRI has own ID for the object (remote id), while NOC assigns its own (local id). So mapping is the relation between NOC's ID, and the pairs of (Remote System, Remote ID).

getmappings API allows to query objects by local or remote id and to get all known mappings in the single call.

Protected Fields

Some fields in UI can be marked as protected, preventing manual user changes.

Development Process Changes

Towards Monorepo

We're beginning to collect all NOC-related repositories to a single repo. Following repos have been merged:

  • collections
  • noc-sync-bind

Flake8 Checks

Codebase is clean enough to enforce previously disabled flake8 check, so they are mandatory now:

  • F403 - 'from xxx import *' used; unable to detect undefined names
  • F405 - 'xxx' may be undefined, or defined from star imports: xxx
  • W605 - invalid escape sequence 'x'

Pending Deprecations

CLIPS-based configuration validation is not supported on python3 installations and will be removed in NOC 20.2. Please select Python 2 if you have config validation rules and consider to rewrite them to ConfDB validation queries or your installation may be stuck on NOC 20.1 release.

New features

MR Title
MR811 Add Task Monitor
MR2753 Add Open/Close Events from Thresholds
MR2952 noc/noc#1150 Add protected fields to web application backend.
MR3063 Lazy MIB loading
MR3082 noc/noc#1221 snmp display-hint processing
MR3090 noc/noc#1110 Add composite connection.
MR3103 noc/noc#1111 Add combo field to objectmodel for Combined connection type.
MR3115 Add single ddash for interfaces in card
MR3179 Rust infrastructure
MR3214 #1219 Additional topology information in managedobject datastream
MR3225 nbi: getmappings API
MR3235 Interface type 'internal'
MR3297 #1264 confdb: meta matchers section
MR3307 #1269: Physical port to logical interface binding
MR3311 installation_id and globally-unique bi_id generation
MR3312 #1255 FM Pools
MR3324 #1277 asset: Serial numbers checking
MR3335 ./noc inventory find-serial command

Improvements

MR Title
MR2991 Fix Cisco Nexus version detection
MR3006 Fix Eltex.LTE.get_inventory script
MR3040 managedobject datastream: admin_status field
MR3067 mib: Resolve syntax references
MR3070 Speedup find service on etl load.
MR3072 ./noc sync-mib speedup
MR3073 Add extra_vars params to pm dash
MR3074 ./noc compile-mib is moved into ./noc mib make-collection
MR3075 Allow to import MIBs in BIG5 encoding
MR3088 inv.objectmodel remove renderer on protocols column
MR3099 Add query param to confdb command.
MR3100 Rebuild datastream when object or objectconnection change.
MR3110 Add make collection from directory to mib command. Fix decode error on compile mib API.
MR3122 Add iface threshold profile handler
MR3139 Interface threshold handling fixes
MR3148 noc/noc#1085 Confdb add VRRP. initial version. Add cisco, huawei
MR3158 Add queue params to close_tt and add_comment method on TT adapter.
MR3159 noc/noc#1184 IGetAlarms SA interface.
MR3166 Fix ldap auth for ldap3 verison 2.0.9-2.6.1 (Py3)
MR3168 Add promote_affected_tt param to AlarmEscalation config.
MR3169 noc/noc#1245 Move configvalidation umbrella to 'Config | Object Policy Violation' AlarmClass.
MR3171 Add escalate umbrella alarm.
MR3187 Update Search by MAC in MO Inteface
MR3202 #1203 Span.set_error, .set_error_from_exc helpers
MR3206 Replace get_handler on classifier service.
MR3215 ./noc datastream rebuild --jobs
MR3217 Add docs for optical model interface
MR3220 Add docs for cpe part of asset model interface
MR3232 #1263 bi managedobject: full link methods detalization
MR3234 test snmp_metrics json
MR3236 noc/noc#1087 ConfDB. add meta chassis_id
MR3238 JUnit XML test report
MR3242 check-labels: JUnit XML report
MR3244 ConfDB: vrrp. add preempt
MR3248 pymongo 3.10.1
MR3253 rust project label
MR3254 mongoengine 0.19.1
MR3255 noc/noc#1078 datastream: config section of managedobject datastream
MR3262 flake8 JUnit XML report
MR3279 Reworked collection JSON sharing
MR3293 inv.objectmodels collection tests
MR3295 Add 'Config | Policy Violations' alarm class.
MR3304 box: move asset discovery just after config one
MR3305 Fix service mailsender
MR3309 test collections' uuids for uniqueness
MR3313 check-labels: Fix migration label detection
MR3314 Retry check-labels as the labels will be fixed from webhook
MR3320 Activate collections tests
MR3322 add new report - load metric max
MR3323 Fix report load metrics
MR3328 check-labels: Detect collections label
MR3329 DCS: Stop one-time resolvers
MR3330 #1191 scheduler: Gracefully apply pending bulk operations on shutdown
MR3333 managedobject datastream: Interface services
MR3339 Catch IOError exception on beef load.
MR3353 Raise error when OID tree from beef end.
MR3354 Add display_hints options to get_table script method.
MR3355 Restore ujson-x 1.37
MR3366 #1278 sae: Pass attributes section

Bugfixes

MR Title
MR2879 Fix Exception when send message 'ServiceStub' object has no attribute 'pub'
MR3013 #1186 py3: Fix request.body processing
MR3014 #1185 py3: Fix error reporting on missed custom modules
MR3015 Fix cleanup 0192_thresholdprofile migration.
MR3019 Fix 0153_managedobjectselector_profile migration after fix 0150_managed_object_profile migration.
MR3024 noc/noc#1195 Catch ValueError when convert_interface_name on TopologyDiscovery.
MR3024 noc/noc#1195 Catch ValueError when convert_interface_name on TopologyDiscovery.
MR3029 py3: alnum_key() for reliable sorting
MR3034 Fix TypeError on http client when working over proxy.
MR3037 #1193 py3: Fix MRT service
MR3041 #1201 py3: Fix http client eof_mark handling
MR3045 #1196 py3: Fix location handling
MR3047 #1209 Fix syslogcollector/trapcollector port binding
MR3055 Fix CH objectprofile datasource bool field.
MR3062 noc/noc#1197 Fix encoding on interfacepath card.
MR3064 py3 fix beef import encoding.
MR3066 Add getting Job log to SA Monitor application
MR3068 fix Alstec, Eltex pon/rg, qsw8200, dlink get_inventory
MR3069 py3: Fix sync-mibs
MR3078 Fix typo in text parse_table on row_wrapper param check.
MR3093 fix ReportAlarmDetail
MR3094 noc/noc#1188 Fix memory leak on SSH CLI.
MR3097 noc/noc#1096 Fix correlator trigger handler call.
MR3104 noc/noc#1213 py3. Fix TypeError when collected config convert to binary.
MR3137 Fix MAC AutoAPI doc builder
MR3143 Fix object datastream iter_changed managed_object check.
MR3152 Fix object iter_changed_datastream typo.
MR3161 Fix Segment MAC Discovery when the object links to itself
MR3162 Fix interface_flap field on ReportMetrics.
MR3178 Fix import __future__ print in deploy scripts
MR3184 Fix promote_affected_tt param when escalation item.
MR3201 sa.managedobject: Remove duplicated LAG Profile column
MR3205 Fix delay escalation when start time < now time
MR3207 #1227: Filter out duplicated metric measurements
MR3221 Rebuild credential cache when change ManagedObjectProfile beef_storage and beef_path_template field.
MR3224 py3: Fix nbi service logging
MR3230 Fix typo in etl extractor.
MR3239 Fix Test notification group, when id > 9.
MR3247 Add ldap3 escape filter chars
MR3250 #1267 Fix Peer.save()
MR3258 py3: Fix whois client
MR3259 #1210: Better json content type handling
MR3263 Add MO queue for maintenance escalation
MR3267 Fix Alarms class filter
MR3288 Fix VRF type name to lowercase (were changed on interface).
MR3291 Add raise_privileges and access_preference params when call login script on suggestcli discovery.
MR3303 inv.objectmodels: Fix duplicated uuids
MR3327 Fix typo max_serial_size on asset discovery.
MR3356 py3: Fix cli.object_stream
MR3358 py3: Fix name encoding when calculate config diff.
MR3359 Convert cmd_next and cmd_stop to bytes on object_cli parser.
MR3371 #1280 py3 ConfDB: Fix node sorting
MR3373 py3: Fix adaptive prompt pattern
MR3374 py3: Fix telnet sb/se logging
MR3376 py3: Clean ETL loader stream from unreadable characters
MR3379 Add fm_pool to field managed object loader.
MR3381 py3: Fix Selectors over attributes
MR3384 Fix NRI discovery

Code Cleanup

MR Title
MR3036 noc/noc#1198 Fix AlarmDiagnostic.get_diagnostics encode.
MR3053 Update docs/src/en/services/icqsender.rst
MR3060 Fix activator.http_get decode error.
MR3061 noc/noc#1197 Fix sortdict on managedobject card.
MR3071 py3. Fix MIBAPI service.
MR3102 LLDP-MIB render test
MR3107 Update requirements/deploy scripts for py3/py2 compat
MR3138 Links to MR list
MR3195 py3. Not encoding utf-8 on geocodercache.
MR3200 noc/noc#1250 Fix ETL for py3.
MR3204 noc/noc#1256 Fix future.set_exception_info on threadpool for py3.
MR3208 Remove 20.1 deprecations
MR3209 Bump pynsq version to 0.8.3
MR3216 Code formatting
MR3219 Fix bytes on script super_username submit.
MR3233 py3: Fix bi extract
MR3249 flake8: Enable W605 check
MR3251 rust: remove "extern crate"
MR3252 CI: Remove migrate stages
MR3260 Remove deprecated models and applications
MR3261 RCA neighbor cache is default and only
MR3264 ujson 2.0.2
MR3285 Merge back collections/ to main repo
MR3286 Restore tests
MR3296 flake8: Enable F403 and F405 checks
MR3310 inv.connectiontypes UUID uniqueueness test
MR3315 Fix cr_context on model
MR3316 Replace gender for c14 connection type part1
MR3317 Replace gender for c14 connection type part2
MR3318 Replace gender for c14 connection type part3
MR3319 Replace gender for c14 connection type part4
MR3336 py3: mailsender fixes
MR3338 Fix typo on asset discovery logging.
MR3340 Replace StringIO to BytesIO on DetailReport for xlsxwriter work.
MR3350 py3. Fix expected byte trace on interface-profile command.
MR3352 py3: Convert rogue_chars to binary type
MR3357 py3: Fix if between None and int type.
MR3362 py3: Disable CLIPS parsers.
MR3365 Clean data type on Consul config client.
MR3372 py3: Replace split_alnum to alnum_key on matchers.
MR3375 py3: Replace decode method to codecs on profiles.
MR3377 py3: Convert addr and port to bytes on http client.
MR3378 py3: Convert hash on geocodercache to text.

Profile Changes

Alcatel.7302

MR Title
MR3089 Alcatel 7302 fix port_map
MR3123 fix Alcatel 7302 serial
MR3243 Alcatel.7302 fix empty b_serial, b_revision. fix port_map. version 2

Alsitec.24xx

MR Title
MR3119 noc/noc#1181 Alstec.24xx. Fix unknown serial number, and devices with 4413 DeviceID.
MR3128 noc/noc#1181 Alstec.24xx.get_inventory. Fallback to CLI when unknown SNMP part_no.
MR3144 Alstec.24xx.get_inventory. Set script prefer to S.
MR3144 Alstec.24xx.get_inventory. Set script prefer to S.

Cisco.IOS

MR Title
MR3009 Update Cisco.IOS profile to support old hardware
MR3173 Cisco.IOS.get_lacp_neighbors, get_version. Small fixes encoding.
MR3185 Cisco.IOS.get_lacp_neighbors. Fix rx_sys_id regex.
MR3229 Cisco.IOS.get_version and get_capabilities cleanup code.
MR3229 Cisco.IOS.get_version and get_capabilities cleanup code.
MR3237 Cisco.IOS.get_switchport. Fix getting vlans from SNMP for python3.
MR3342 Cisco.IOS.get_cdp_neighbors. Replace OID to mib name.
MR3351 Cisco.IOS.get_inventory. Use matchers instead match.

DLink.DxS

MR Title
MR3091 Bypass confirmation of firmware upgrade on DGS-1100-10/ME
MR3363 DLink.DxS. Fix decode on profile.
MR3367 Dlink.DxS.get_interfaces. Remove decode ascii interface description.

Dahua.DH

MR Title
MR3142 Dahua.DH. Fix getting config from RVi web cam.
MR3174 noc/noc#1216 Dahua.DH.get_version. Fix script on cam without PTZ module.

Eltex.MES

MR Title
MR3155 add_metrics_SensorStatus_Eltex_MES
MR3177 fix_Environment_Sensor_Status_Eltex_MES

Eltex.MES24xx

MR Title
MR3007 Fix Eltex.MES24xx.get_lldp_neighbors script
MR3164 fix_get_lldp_neighbors_Eltex_MES24xx

Generic

MR Title
MR3020 Generic.get_lldp_neighbors. Strip x00 symbol on SNMP string.
MR3023 Generic.get_capabilities. Use cache value sysObjectID for getting EnterpriseID.
MR3023 Generic.get_capabilities. Use cache value sysObjectID for getting EnterpriseID.
MR3032 Generic.get_beef. Fix fsm collecting - not collected if get_version first.
MR3084 Generic.get_capabilities. Fix bad enterprise id check.
MR3149 Generic.get_beef. Catch SNMP error when collected beef.
MR3218 py3. Message attribute not supported on Exception.
MR3348 Generic.get_switchport. Fix getting vlans on SNMP.

H3C.VRP

MR Title
MR3240 H3C.VRP. Fix get_version and add SNMP scripts.
MR3300 H3C.VRP.get_version. Fix regex for S3100-8TP-EI model.

Huawei.MA5600T

MR Title
MR3022 Huawei.MA5600T. Fix lstrip on profile parse_table.
MR3163 noc/noc#1226 Huawei.MA5600T. Add pattern_more for MA5800-X2 platform.
MR3163 noc/noc#1226 Huawei.MA5600T. Add pattern_more for MA5800-X2 platform.
MR3337 Huawei. Disable CLI metrics.

Huawei.VRP

MR Title
MR3105 Huawei.VRP. Add S3328 to hw_series.
MR3113 Huawei.VRP. Fix version and inventory script for CX300 platform.
MR3116 noc/noc#1163 Huawei.VRP.get_inventory. Add S8504 slot count.
MR3126 noc/noc#1163 Huawei.VRP. Add error_pattern_prompt and fix get_version encoding.
MR3345 Huawei.VRP.get_switchport. Fix getting port vlan on SNMP.

Iskratel.MSAN

MR Title
MR3344 Fix send_on_syntax_error params for py3.

Juniper.JUNOS

MR Title
MR3058 Juniper.JUNOS.normalizer. Fix detect vrf type.
MR3256 add snmp oid jnxVirtualChassisMemberMacAddBase. fix get_chassis_id Juniper_JUNOS

Qtech.QSW2800

MR Title
MR2960 Add support for stack in QSW-8200-28F-AC-DC
MR3222 Qtech.QSW2800.get_capabilities. Fix not supported operation on 'NoneType' and 'int' type.
MR3343 Qtech.QSW2800.get_fqdn. Fix py3 pattern encoding.

SKS.SKS

MR Title
MR3127 noc/noc#1180 SKS.SKS. Fix get_spanning_tree designated_port_id and get_inventory unknown E1 module.

Siklu.EH

MR Title
MR3364 Siklu.EH. Fix cleaned_input to binary data.

rare

MR Title
MR3038 Add NSN.hiX56xx.get_inventory script
MR3050 DCN.DCWL. Fix profile for SNMP
MR3052 DCN.DCWL.get_version. Fix typo on matchers.
MR3079 Carelink.SWG. Fix pattern_more on profile.
MR3083 Qtech.QSW8200.get_lldp_neighbors. Add ifName to port_type.
MR3108 fix_get_inventory_Eltex_ESR_profile
MR3150 fix_AlliedTelesis.AT9900_profile
MR3160 add_get_ifindexes_AlliedTelesis.AT9900_profile
MR3167 add_get_lldp_neighbors_AlliedTelesis.AT9900
MR3180 add_cli_get_lldp_neighbors_AlliedTelesis.AT9900
MR3186 add_get_capabilities_AlliedTelesis.AT9900
MR3196 Fix get_interfaces for BDCOM.xPON profile
MR3241 Fix invalid JSON
MR3245 Fix get_interfaces NSN.TIMOS
MR3298 fix_get_interfaces - port-channel no has iftype
MR3306 NSN.TIMOS Fix get_intefaces script
MR3308 Iscratel.ESCOM fix profile
MR3332 Iskratel.ESCOM fix lldp neigborg script
MR3346 py3. Cleanup get_lldp_neighbors script for SNMP.
MR3360 fix_get_config_NSN.hiX56xx
MR3361 fix_pattern_more_Huawei_VRP3
MR3369 Fix Check metrics collect on DCN and Rotek profiles.
MR3370 py3: Convert telnet_send_on_connect to binary.
MR3382 NSN.TIMOS.get_interfaces. Fix empty mac address on tunnel interface.