NOC 20.2
20.2 release contains
249
bugfixes, optimisations and improvements.
Highlights
Python3 Compatibility
As it was declared in
Python3 Compatibility Statement<release-19.4-py3>
, 20.2 is the first
Python3-only release. This allows us to clean up and optimize code base
in following ways:
All Python2 compatibility layers are removed.
Key dependencies (Tornado, Django, etc) are upgraded to latest
stable versions.
async/await are used instead @tornado.gen.coroutine
in all the places.
Type annotations are heavily used in newly-introduced and in
modified code.
Low-level network-handling code, including SNMP, RPC, HTTP-client and
all CLI variants are rewritten from Tornado to Python3-native asyncio.
It allows to clean up code, fix some old cryptic bugs and introduce
gentler error- and timeout- handling.
NOC 20.2 is compatible with Python 3.7 and 3.8.
So we have to admit the Python3 migration Odyssey is over. It tooks over
one year, consumes lots of work, sometimes with acrobatic exercises to
maintain compatibility and stability. But in addition to code
simplicity, stability and instrumentation we gave serious performance
benefits. Our investigations on real-world installation shows that NOC
20.2 requires 10-15% less CPU, than previous Python 2 based releases.
Interface Description Topology Discovery
Meet the ifdesc
topology discovery
method. It is last-resort semi-automatic method of linking based on
interface description analysis. ifdesc
comes to resque when all other methods failed. Operator has to define a
set of regular expressions to extract a neighbor information from
descriptions and has to configure network equipment. In contrast to
manual linking, ifdesc
is configurable as
common topology discovery method. Its precedence over other methods can
be adjusted, so its results may be refined by more precise automatic
methods (i.e. LLDP), and it can be used to refine more error-prone
methods (i.e. MAC).
ifdesc
is the 14th topology discovery
method available out of the box.
MAC Blacklist
Costs cutting and lack of proper Q&A sometimes leads to large parties of
network equipment with non-unique MAC addresses. Overall impact may
vary: from insignificant, to ARP-cache poisoning, to inpredictable STP
topology changes. It-also affects the topology discovery methods based
on neighbor identification upon MAC address (LLDP, STP, MAC). So its
safer to maintain a list of broken MAC addresses to notify personnel on
possible problems.
MAC blacklist is maintained and distributed on deploy, like other
collections, and can be adjusted and maintained locally and shared back
to the repo.
LLDP discovery method now consults on MAC Blacklist and tries to
mitigate problem by using neighbor information from additional TLVs.
RCA Downlink Merge Policy
Real-world networks often repeats the pattern when aggregation switch
(A) is placed on PoP with redundant reserved power and connected to home
access switches (B, C, D, E), which are powered from non-redundant city
power grid.
graph TD;
A --> B; A --> C; A --> D; A --> E;
During the maintenance on power grid, several access switches may expect
power loss, while aggregation switch stays on-line as unaffected or
switched to reserve power. Current topology-based RCA scheme will
register each access switch failure as separate root cause alarms and
will propagate them as separate trouble tickets. It may take a time for
personnel to detect all those problems are related and inferred by power
problems.
NOC 20.2 introduces additional Downlink
Merge
policy. Downlink Merge
is
enabled on aggregation switch (A) and allows to tie all direct children
alarms together and to process and escalate them as a one failure.
Network personnel will take a hint to check alarms as related together
and will be notified to check the power.
Datastreams can be configured to separate views, filtering or morphing
records. Views, called formats
can be
used to enrich datastreams with data from external sources, or to
restring amount of data to be passed to external systems.
Each format
can be configured to
additional API Key role, allowing granular access.
Customized SNMP-response parsers
Broken SNMP implementations are in the wild. Sometimes responses are
malformed, while still containing meaningful data. So we added Profile.get_snmp_response_parser()
method to
override default response parser to customized one when necessary.
New features
MR
Title
MR3418
Grafana: Add new DOM dash
MR3550
Add Ignore Pattern func
MR3600
#1287: Multi-Format DataStream
MR3632
#1086 IfDesc topology discovery
MR3653
#1301: MAC Blacklist
MR3668
#1222 RCA 'Downlink Merge' policy
MR3746
Profile.snmp_response_parser
Improvements
MR
Title
MR3392
noc/noc#1283 Add allowDiskUse to reportdatasources.
MR3407
Add combo connection type to Huawei Quidway series.
MR3428
Add Combo port to Raisecom models.
MR3442
http-parser 0.9.0
MR3444
yield from
MR3445
pytest 5.0.3
MR3461
Fix some models
MR3470
datastream: ManagedObject.object_profile.tags
MR3476
Fix report max load metrics 2
MR3490
Deduplicate Datastream Changes.
MR3491
Replace imp to importlib on mib service.
MR3494
Add SubscriberProfile to ETL loader
MR3500
ThreadPoolExecutor: Support for context protocol
MR3501
Update verify-models script
MR3512
Register Datastream Changes when Service - Interface binding changes
MR3516
Fix get_interfaces for DES-3200/C1
MR3522
Add errors_in_sum, errors_out_sum column to reportmetrics.
MR3524
fix_objectmodels_comboports_eltex_mes
MR3527
nbi Path API: Specify start by OFM order_id
MR3533
Bump requirements versions
MR3539
Django 3.0.5
MR3542
Add iface_hints to RemoteSystem portmapper.
MR3553
Tornado 6 support
MR3554
NetworkX 2.4
MR3555
Make Service.subscribe a coroutine
MR3557
Cleanup interface service field when remove service.
MR3560
Add pool to reportdiscoverypoison.
MR3561
Impoved logging when use neighbors discovery cache.
MR3571
Python 3.8 tests, pyproj 2.6.0
MR3580
Add quiet params to etl extract command. Supress exception if set.
MR3581
Add retry escalation to Escalation limit exceeded.
MR3587
add_new_model_Nag
MR3588
fix_objectmodels_eltex_LTP-LTE
MR3591
Add rules to test inv.objectmodels
MR3592
Fix some models
MR3615
Native PeriodicCallback and PeriodicOffsetCallback
MR3617
http client: Asynchronous DNS resolver
MR3619
Add cutomized dashboard to PM.
MR3624
MAC Segment discovery. Check object level on same segment for uplink detect.
MR3625
#1298 PrefixDiscovery takes account of MO's VRF
MR3626
#1299 Hide hidden custom fields from ipam view
MR3627
Django 3.0.6, mongoengine 0.20.0, crontab 0.22.6, redus-3.5.2, hiredis=1.0.1, ipython 7.14.0
MR3633
Add ELtex MIB preference.
MR3636
Add Eltex MIBs.
MR3637
Add catch OIDCollision to mib upload command. Add detect some errors when upload mib.
MR3641
IOLoop.add_callback() cleanup
MR3650
Native asyncio SNMP implementation
MR3657
Add Huawei.MA5600T profile check rules
MR3662
Add Eltex MIBs.
MR3674
Add json_unique_fields to ProfileCheckRule.
MR3678
Add 'Discards In' and 'Discards Out' to Report metrics.
MR3682
noc/noc#1295 Add Cisco ASR1001-X model.
MR3685
Remove Service.ioloop
MR3694
Streamlined SNMP Beef implementation
MR3695
asyncio-native NSQ topic implementation
MR3705
asyncio-native CLI
MR3713
Replace resolve_ifindexes to resolve_properties on InterfaceDiscovery.
MR3716
Fix filling Capabilities from ConfDB.
MR3719
Add dump argument to confdb commands.
MR3747
Add BI Dashboard collection.
MR3751
Raise CLI CLILowPrivileges when no up privileges by enable.
MR3752
Send command_exit when close cli stream.
MR3754
Add get_fqdn to profiles.
MR3760
Add 'Exclude on MAC Black List' param to ReportDiscovery Poison.
MR3763
Add raw_result params to script http methods.
MR3765
Add errors="replace" policy to script parse_object_stream.
MR3768
Change docker image's python to 3
Bugfixes
MR
Title
MR3389
fix report - load metric max
MR3395
Fix save on ModelProtectionProfile.
MR3397
Tgsender: fix py3
MR3400
Fix bi encode error when empty field description.
MR3405
Fix changing objectmodel combo field.
MR3419
Fix AlowDiskUse param on aggregate report
MR3426
Fix DatasourceCache compress encoding.
MR3440
Fix Consul session lost logging
MR3446
Disable smart_text over return http.get json result.
MR3448
Fix comma on mo pm dashboard.
MR3479
py3: Fix DNS RR comparison
MR3483
Add is_managed=False to maintenance escalator handler.
MR3484
Fix detect selected column on ObjectDetail Report.
MR3502
Fix bytes on asset generate_serial.
MR3503
Change enable_link to enable_service on nri service discovery.
MR3507
py3: Fix whois update-cache
MR3513
Add json_unique_fields to mibpreference.
MR3517
Fix scheduler shutdown
MR3518
Fix CH Dictionary split data to chunk.
MR3534
Remove validation policy button from UI.
MR3537
Fix fix-pip on PIP's beta versions
MR3556
Fix SAE return redirect. Was broken on noc/noc!3508.
MR3574
Fix Link.managed_objects on Django3
MR3576
Fix not operand on ConfDB.
MR3612
Catch portmapper exception on NRI discovery.
MR3618
httpclient: Fix EOF handling
MR3620
noc/noc#1283 Fix ReportLinkDetail for py3.
MR3634
Fix trace when export fm.mibpreference collection by command.
MR3640
Fix detect coroutine on Script.on_error_sequence.
MR3646
Fix ReportDiscovery Poison MAC view.
MR3649
Fix script on_unprivileged_prompt when no raise privileges.
MR3651
classifier: Fix bytes processing
MR3664
Fix 'TypeError' on confdb iter_nodes sorted.
MR3666
HTTPClient: Better handling of connection errors
MR3671
chwriter: Fix channel messages calculation
MR3672
Fix race condition during lazy mib loading
MR3673
Lock all RPC deserialization for a better times
MR3676
Fix logger argument on xmac discovery.
MR3692
Fix dictionary keys changed during iteration on extdocapplication.
MR3693
SNMP: Fix GETNEXT raw_varbinds processing
MR3702
Fix confdb protocols applicators for type normalizer.
MR3707
Fix MRT service
MR3708
ETL: Fix parallel extraction for Oracle
MR3710
Fix CLI loop context usage
MR3712
redis: delete_many may be called with empty keys
MR3714
Fix typo in InterfaceDiscovery resolve_properties.
MR3724
httpclient: Fix connection reset handling
MR3729
CLI: Fix keepalive timeout handling
MR3731
py3. Fix get_colors.
MR3749
CLI: Fix connection reset handling
MR3750
Fix typo on MRT service.
MR3753
Fix params on LLDP Discovery logger.
MR3756
snmp: Fix broken error-index
MR3757
Fix Script.on_error_sequence sequence type.
MR3758
Fix typo on MacBlackList js Application.
MR3759
Fix NSQ topic shutdown sequence
MR3762
Fix BER unknown types reporting
MR3769
Fix script.submit debug.
MR3770
Fix catch ConnectionResetError on http_client.
MR3771
Right fix ConnectionResetError on http client.
Code Cleanup
MR
Title
MR3347
py3. Change rogue_chars on profiles to bin.
MR3387
Remove py2 tests
MR3388
Set black target to python 3.6
MR3390
Fix DDash sensorstatus
MR3390
Fix DDash sensorstatus
MR3394
Remove CLIPS config validation
MR3401
py3. Fix compile-bytecode.
MR3417
py3: Remove six.PY2/six.PY3 usage
MR3421
py3: Remove six type aliases usage
MR3439
py3: Remove six.iter* wrappers
MR3447
Remove @six.python_2_unicode_compatible and six.with_metaclass
MR3457
Move to py3-style type annotations
MR3458
Clean up .docker directory
MR3463
Remove six
MR3464
Cleanup noc.core.comp bord/bchr/make_bytes functions
MR3465
Use dict comprehensions
MR3466
Use py3 to build docs
MR3467
Fix docs builder
MR3469
py3: Fix build-go.py
MR3473
Remove CLIPS-based validation models
MR3474
mypy: Remove --py2 flag
MR3508
py3: Remote raise tornado.gen.Return() usage
MR3509
Fix type annotations
MR3519
Source coding specification cleanup
MR3523
py3.8 compatibility fixes
MR3525
Use native time.perf_counter()
MR3526
Remove from builtins import
MR3528
Remove pypy support
MR3529
Ping: Remove io_loop
parameter
MR3531
httpclient: Remove io_loop
parameter
MR3535
DCS: Remove io_loop parameter
MR3536
Remove coding directive
MR3538
Remove deprecated get_absolute_url
method
MR3540
Remove io_loop
parameter from UDPServer
MR3541
Remove io_loop
parameter from CLI
MR3543
Remove PeriodicOffsetCallback and NSQ TopicQueue io_loop parameter
MR3544
Remove snmp io_loop parameter
MR3545
Remove deprecated TracebackFuture alias
MR3546
Remove scheduler IOLoop parameter
MR3593
Cleanup super() call
MR3606
Cleanup tornado.gen usage
MR3614
Fix DOM ddash
MR3628
Remove deprecated get_future_loop() compatibility call
MR3631
Fix import deprecation warning
MR3639
Fix id ddash for Grafana 6
MR3643
Add/new ddash stp delta
MR3659
Fix Radio Ddash
MR3665
Replace socket.error with OSError
MR3726
20.2 deprecations
Profile Changes
Alcatel.7302
MR
Title
MR3594
Alcatel.7302. Add SNMP to profile.
Alsitec.24xx
MR
Title
MR3437
noc/noc#1181 Alstec.24xx. Add platform name normalizer.
Cisco.IOS
MR
Title
MR3767
Cisco.IOS.get_interfaces. Use dict structure instead list.
DLink.DxS
MR
Title
MR3596
DLink.DxS_Cisco_CLI.get_interfaces. Fix description decode.
MR3727
DLink.DxS_Smart.get_interfaces. Fix typo.
MR3737
DLink.DxS.get_lldp_neighbors. py3. Fix remote_port format to string.
Eltex.MES
MR
Title
MR3391
Eltex.MES Update metrics
MR3434
Eltex.MES: Fix get_mac_address_table
MR3441
Eltex.MES: Update profile for 3124/3124F
MR3487
Eltex.MES. Add detected platform to profile.
MR3547
add_new_model_eltex_mes-3116F
MR3679
Eltex.MES.get_capabilities. Add SNMP methods.
MR3680
Eltex.MES.get_interfaces. Use Generic SNMP method.
MR3700
Eltex.MES.get_interfaces. Add Vlan and Lo filter on SNMP.
MR3722
Eltex.MES. Add always_prefer S for 3324/3348 models.
Eltex.MES24xx
MR
Title
MR3697
Eltex.MES24xx. Add config prompt to profile.
MR3738
Eltex.MES24xx.get_lldp_neighbors. Add 'Interface Alias' to port subtype.
Generic
MR
Title
MR3559
Generic.get_chassis_id. Add 00:01:02:03:04:05 to IGNORED_MACS.
MR3565
Generic.get_mac_address_table. Filter vlan 0 number.
MR3573
Generic.get_interfaces. Refactor script for get_interface_properties use.
MR3648
Add Generic.get_fqdn script.
MR3681
Eltex.MES.get_spanning_tree. Add SNMP method.
MR3687
Generic.get_beef. Fix encode method.
MR3688
Generic.get_spanning_tree. Add detect port role.
MR3696
Generic.get_intefaces. Filter equal values on getting portchannel.
MR3706
Add get_interface_properties to get_ifindexes profiles
MR3733
Generic.get_vlans. Fix py3.
MR3734
Generic.get_mac_address_table. Skip unknown interfaces.
Huawei.MA5600T
MR
Title
MR3412
Huawei.MA5600T.get_cpe. Use codecs.encode instead .encode.
MR3582
Huawei.MA5600T.get_interfaces. Add interface hints field.
MR3595
Huawei.MA5600T. Add get_lldp_neighbors script.
MR3654
Huawei.MA5600T.get_portchannel. Add allow_empty_response to 'display lacp' command.
MR3656
Huawei.MA5600T.get_portchannel. Use allow_empty_response on 'display lacp' command.
Huawei.VRP
MR
Title
MR3669
Fix tagged vlan parsing on switch with 40G interfaces
MR3699
Huawei.VRP.get_interfaces. Use show vlan command for switchport info.
MR3718
Huawei.VRP. Fixes ConfDB normalizer.
Iskratel.MSAN
MR
Title
MR3755
noc/noc#1232 Iskrate.MSAN. Change detect version to 'Steer Version'.
Juniper.JUNOS
MR
Title
MR3386
Fix Juniper.JUNOS.get_interfaces script
MR3413
Juniper.JUNOS.get_interfaces. Use codecs.decode insterad .decode.
Orion.NOS
MR
Title
MR3468
fix get_version - space removed in platform
MR3492
noc/noc#1230 Orion.NOS. Add convert_interface_name method.
MR3590
Fix Orion.NOS.get_lldp_neighbors script
Qtech.QSW
MR
Title
MR3599
noc/noc#1231 Qtech.QSW.get_version. Fix version regex.
Qtech.QSW2800
MR
Title
MR3711
noc/noc#1190 Qtech.QSW2800.get_version. Fix 3.0.1-R1-BETA3 firmware.
MR3721
Qtech.QSW2800. Fix patter_error for QSW-3470-10T with 3.0.1-R1-BETA3.
MR3766
noc/noc#1190 Qtech.QSW2800.get_lldp_neighbors. Add port description and sysname
Raisecom.ROS
MR
Title
MR3399
Raisecom.ROS.get_version. Add detect Qtech vendor.
MR3566
Raisecom.ROS. Add fastethernet1/0/1 to management for ISCOM26(?:24|08)G.
MR3735
Raisecom.ROS. Add SNMP to scripts.
MR3736
Raisecom.ROS.get_capabilities. Fix STP detect.
SKS.SKS
MR
Title
MR3601
noc/noc#1180 SKS.SKS.get_interfaces. Fix tagged vlan none on command.
Siklu.EH
MR
Title
MR3610
Siklu.EH.get_mac_address_table. Fix mac type detected if quota column end.
Zyxel.ZyNOS
MR
Title
MR3689
noc/noc#1290 Zyxel.ZyNOS. py3, Convert pattern_zynos prompt to binary.
MR3720
Zyxel.ZynOS.get_inventory. Disable keep_cli_session after script.
rare
MR
Title
MR3393
Eltex.MES5448 update profile
MR3410
py3. Cleanup OID iface MAC to IF-MIB::ifPhysAddress name.
MR3420
Generic.get_mac_address_table. Fix getting mac address tables by SNMP.
MR3450
DCN.DCWL.get_metrics. Fix empty mac address.
MR3453
Convert command_submit on self.cli to bytes.
MR3454
Qtech.QSW.get_chassis_id. Add render_mac to SNMP method.
MR3459
py3. Fix convert hex values on profiles.
MR3495
noc/noc#1175 Huawei.MA5300.get_version. Add MA5303 to detecting profile.
MR3725
HP.ProCurve. Cleanup py3.
MR3739
DCN.DCWL.get_metrics. Fix rtt check.
MR3748
Qtech.BFC_PBIC_S. Use strict snmp response parser.
MR3764
Vitesse.VSC. Fix profile pattern_more.