NOC 19.4

In accordance to our Release Policy we’re proudly present release 19.4.

19.4 release contains 386 bugfixes, optimisations and improvements.

Highlights

Python3 Compatibility Statement

Python3 compatibility was one of primal focuses during 19.4 release cycle. We made lots of code reviews and cleanups to improve py3 compatibility. NOC 19.4 is able to perform test suite both on Python 2.7 and Python 3.6 and most of functions work on py3, though some glitches still remain. So we continue routine work on reaching full py3 compatibility.

We expect that:

  • NOC 20.1 will have full py3 compatibility and will allow to choose between py2 and py3 during deploy.

  • NOC 20.2 will by py3-only release with all py2-compatibility stuff removed.

  • NOC 20.3 will use py3 features like async/await and will switch from tornado to FastAPI.

Extended MAC discovery

Discovery got extended MAC (xmac) discovery method, showing good results in linking dumb terminal equipment like CPE’s, IP Phones, IP cams, which have no other reliable discovery methods.

See #1071 for details.

DataStream improvements

IPAM got VRF, Prefix and Address streams allowing to expose all IPAM data to external systems.

managedobject DataStream got additional tags attribute exposing ManagedObjects’s tags. Tags allow to pass unstructured metadata to external system.

managedobject DataStream also exposes comprehensive VPN information.

nbi improvements

NBI got 3 additional groups of API:

  • objectstatus for online Managed Object status requesting

  • config and configrevisions for accessing Managed Object’s config

  • path - the swiss-army knife for network topology paths calculation

time_delta in metrics

Discovery writes time elapsed from previous check to time_delta field, allowing to store metrics deltas instead of speed, allowing to perform tasks like traffic calculation and robust error detection.

Compare Configs

Configs from different managed objects can be compared via UI

BI improvements

Amount of reboots, outages, STP topology changes are added to Managed Objects’ BI statistics allowing additional problem drill-down vectors.

ICQ sender

Additional icqsender service allows to push notifications via ICQ messenger.

Development Process Changes

py3 tests

Python3 tests are mandatory now and all test suite must pass python3 testings stages as well.

Mypy integration

NOC adopts Mypy. Though mypy tests are still advisory developers are urged to add type annotations to new and to the refactored code.

Code Formatting

NOC adopts Prettier for JS, JSON, CSS and YAML code formatting. CI pipeline checks code formatting of changed files. Any misformatting considered the error and CI pipeline fails at the lint stage. We recommend to add Prettier formatting to git’s pre-commit hook or to the IDE’s on-save hook.

Docker container is also available. Use:

docker run --rm \
    -w /src \
    -v $PWD:/src \
    registry.getnoc.com/infrastructure/prettier:master \
    /usr/local/bin/prettier --config=.prettierrc --write <file name>

to format file

New features

MR

Title

2234

network segment: search & select object

2374

Took default user password and email from env on initial migrations

2532

nbi: config and configrevisions API

2613

metrics: Collect time_delta parameter

2655

DataStream: ManagedObject tags

2671

Config diff from two objects’ revisions

2714

noc/noc#1126 Add n_reboots, availability, total_unavailability, n_outages,…

2727

Path API

2764

DataStream: Expose VPN information to managedobject datastream

2772

Add ICQ sender notification service

2862

Extended MAC discovery

2881

datastream: IPAM integration

2886

nbi: objectstatus API

Improvements

MR

Title

1787

Add full address on map

1990

Replace pylibssh2 with ssh2-python

2408

docs: Fix profile anchors

2415

ConfDB: Syntax documentation

2492

Add discovery status classificator to report object stat.

2504

Check for nbi MR label

2506

cmtext height 100%

2508

Filter enable_discovery False from Report Profile Check Summary.

2533

login: Log authentication failures

2548

Change to Edit in ModelApplication.js title

2562

translate syslog

2567

Set secure and httponly cookie

2571

prepare for build

2574

Prettier code formatting check

2575

pynsq 0.8.3

2576

Django 1.11.24

2577

pyfilesystem2 2.4.10

2591

NSQ http publisher

2594

escalation: Configurable group tt promotion

2598

Add test_prefixlist.py

2602

Add IPv4 address format check to ManagedObject etl loader.

2609

Optimize uplink calculations for complex topologies

2614

#955 sa.managedobject: Group editing of description

2625

ClickHouse: Insert into JSONEachRow format

2627

python-consul 1.1.0

2636

Add ignore_empty_cli_response to script cli method.

2638

Add loader to GeocoderCache.

2656

sa.managedobject: Additional access checks in the config preview

2669

Cleanup CH sharding code

2673

JSONL format for BI ETL

2693

Add ObjectProfile field to Alarm and Reboot BI models.

2712

Add arguments to BI extract method.

2717

Fix the name of the standard for Gigabit Ethernet.

2718

pytest 4.6.6, Coverage 4.5.4, removed unused option ./noc test –test-report

2719

typing 3.7.4.1

2722

Extend logging on MIB import.

2730

sa.managedobjectprofile, inv.interfaceprofile: Case-insensitive substring search

2742

maintenance networksegment type changed from lookup to tree

2754

MO profile, numberfield minValue set

2762

ConfDB: Data Types

2763

ConfDB: MPLS protocol

2791

RepoPreview style fixed

2799

Add ICQ sender service docs

2813

modelfilter.List remove padding: 10

2823

Add segment mac to discovery command.

2824

Add forwarding_instance index to subinterface model.

2825

Add config.version_format for noc version customization.

2829

Extend logging when ConsulDCS lost session.

2846

Fix icqsender doc

2855

Add SNMP encode_oid function speedup.

2871

Add ObjectProfile fieild to ManagedObject BI model.

2889

fix duration for active alarms

2892

CI: Run python3 tests automatically

2906

Speedup ReportAvailability.

2908

Add url storage path to beef command

2921

noc/noc#1159 Add eof_mark param and extend logging on digestauth.

2924

noc/noc#1157 Add filter_vendor, filter_platform, filter_version to ManagedObjectSelector application.

2929

Add new get inventory2

2934

Add description param to beef collect command.

2938

Add ad-hoc beef options to script

2940

PyFilesystem2 2.4.11

2943

noc/noc#1104 Add static MAC to mac discovery.

2947

Update HP.ProCurve profile

2958

noc/noc#1158 Disconnect connections on asset discovery.

2959

py3: Set py3 tests as mandatory

2962

Replace interface profile on nri_service discovery when change settings.

2964

noc/noc#1088 Add ifindex and mac to confdb interface meta.

2969

Add to beef command allow use local specs.

2972

Add ERR_SNMP_BAD_COMMUNITY error when credentials without SNMP.

2979

noc/noc#1098 Remove read() from read json file in inject-event command.

2997

Speedup DiscoveryTopology Problem report.

3001

Use sys.platform unstead os.uname().

Bugfixes

MR

Title

2281

Fix HP.ProCurve.get_lldp_neighbors script

2446

Add DISABLE_SERVER_SIDE_CURSOR params to connect django db.

2463

form objectvalidationpolicy fixed

2464

Fix KB index page trace when empty user history.

2481

Remove newline from version info

2486

Fix interface-profile command trace when connect.

2490

Fix ddash when no metric on interface profile

2501

Fix matcher parsing

2503

Ensure User.last_login is nullable

2535

inv.map right width increase

2536

Fix on_init attribute on administrative_domain model.

2545

Use ReportModelFilter for ReportDiscoveryLinks.

2547

Change user_permissions field to permissions (fix create user).

2549

Add ch_escape to managedObject BI extractor.

2558

noc/noc#1095 Fix mongo connect on commands job, run, wipe.

2559

Fix DiscoveryResult reportdatasource high memory consumption.

2560

Register unknown part_no use first vendor_code

2568

Add object_profile field to managed_object iter_datastream.

2570

translate Add Insert for listformfield and gridfield fixed

2572

Add use_mongo to mrt services.

2587

#1105 Fix apply fix_object_uplinks and fix_alarm_managedobjectprofile

2595

Fix ReportObjectDetail append row.

2600

#1100 Fix FirmwarePolicy.get_recommended_version()

2612

collections: Ignore unknown fields

2618

Update commands/whois.py

2621

Fix mongo connect on whois command.

2622

Fix clear unknown model when create.

2629

Add UUID to JSON Unique field on Inventory models.

2634

#1112 fix csv-import/export command

2637

Fix interface_flap param on ReportMetric.

2660

Fix broke –clean argument on events command

2666

Fix ‘code’ object is not callable on metric shard function.

2696

Fix syslogcollector.register_message typo.

2699

requirements tablesorter & table2csv added

2700

#1122 Split too large published messages into parts

2710

Fix CSV Import

2713

noc/noc#1122 Workaround for stuck metrics queue when message very large.

2716

Fix _iter_metrics_raw_chunks chunk limit config.

2747

Handle NSQ publisher failures

2765

Fix escalation Stop Processing

2766

Set availability ManagedObject BI extractor to 100%.

2773

Chrome ignores autocomplete=’off’ - fixed

2798

Catch SyntaxError trace when managedObject card open.

2800

NBI objectmetrics. Return 404 if requested ID not in system.

2800

NBI objectmetrics. Return 404 if requested ID not in system.

2811

whois: Do not update cache on download errors

2822

Add batch work to fix_link_all_objects.

2832

DataStream client: Handle DCS ResolutionError properly

2832

DataStream client: Handle DCS ResolutionError properly

2836

Fix service shutdown on nsq topics shutdown timeout

2845

noc/noc#1139 Fix _write_int on ber.pyx for 64-bit value.

2851

BBox checking

2876

Fix initial_data[“pool”] trace when managedobject change Pool.

2880

Add cert param to ConsulHTTPClient (addition version 1.1.0).

2887

Fix escalation closed while escalated proccessed.

2942

noc/noc#1094 Add drop old maintenance collections.

2956

noc/noc#1155 Catch LDAPCommunicationError and LDAPServerPoolExhaustedError LDAP server error.

2957

Fix typo in xmac discovery.

2966

Add ValueError to catch cards confdb error.

2973

Fix Invalidate credentials cache when object_profile change.

2978

noc/noc#1153 Fix config value if consul set it to empty string “”.

3003

Fix UserProfile import on wipe command.

3005

#1102 Fix RouterOS tokenizer

Code Cleanup

MR

Title

2325

fix_mib_make-cmib

2445

docs: Source code documentation build

2480

Start NOC 19.4 release cycle

2505

move lib/convert.py to core/convert/dbm.py. fix profiles

2514

move lib/dateutils.py to core/dateutils.py.

2515

move lib/escape.py to core/escape.py

2516

move lib/timepattern.py to core/timepattern.py

2537

move lib/geo.py to core/geo

2544

move lib/url to core/url

2585

move lib/forms to core/forms

2589

move lib/rpsl to core/rpsl

2601

move lib/prettyjson to core/prettyjson

2611

move lib/text to core/text. add tests

2615

bump bootstrap to 3.3.7 cause of web.json already have one

2617

move lib/validators to core/validators

2626

remove lib/url.py

2654

Add migration statements for release notes

2709

Fix Clear Alarm, when option disabled in alarm class

2774

Remove deprecated config section

2878

Geocoders moved to noc.core.geocoder

2893

py3 compatibility fixes

2894

py3: Fix base64 usage

2895

py3: zlib fixes

2896

py3: Replace logger.warn() with logger.warning()

2897

Profile re r cleanup part1

2898

py3: hashlib fixes

2899

py3: Fix print usage

2900

Cleanup app file on re.

2901

Profile re r cleanup part2

2902

py3: BER fixes

2903

py3: Fix CH charding

2905

Remove NOC 19.4 deprecations

2907

py3: Fix md5crypt

2909

py3: Fix fm_escape/fm_unescape

2912

py3: Fix cmp() usage

2913

Fix Generic.get_metrics script

2916

py3: unicode() fixes

2918

Profile re r cleanup part3

2923

dict() optimization

2925

py3: GridVCS fixes

2926

py3: SNMP Fixes

2928

Fix typo on smart_text argument position.

2930

py3: Fix ConfDB

2931

py3: Fix SNMP TC processing

2932

py3: Fix model unicode test

2933

Cleanup regex errors on profile.

2936

py3: Fix telnet

2939

py3: unicode check fixes

2941

py3: Fix DNS

2944

py3: unicode check fixes.

2948

py3: Fix cli

2949

py3: Fix BasicAuth test

2950

py3: Fix NSQ mpub

2954

py3: Customized pyclips

2955

py3: Fix main.desktop test

2963

py3: Fix replace_re_group

2967

Fix \x symbol on strip.

2977

Fix ‘Overlong 2 byte UTF-8 sequence detected when encoding string’ errors on activator methods.

2982

py3: Various fixes

2984

py3: Translation fixes

2985

py3: Login service fixes.

2986

py3: config string params fix.

2987

py3: Fix pickle field.

2990

py3. Fix beef working.

2992

Fix limit param check on extapplication.

2993

py3 fix chwriter records type.

2994

Py3. Fix build_echo_request.

2996

Fix cleanup-pyc.

3002

Fix Django version on docker requirements.

Profile Changes

Alsitec.24xx

MR

Title

2590

Update Alstec.ALS profile

2784

Alstec.24xx.get_interfaces. Add name port format.

2796

Alstec.24xx. Add get_inventory script.

2884

Fix prompt in Alstec.24xx profile for exclude banner

Axis.VAPIX

MR

Title

2491

Add Axis.VAPIX confdb normalizer.

2500

Fix Axis.VAPIX profile API parser when comment in output.

2520

Axis.VAPIX. Fix normalize_resolution in confdb normalizer.

BDCOM.IOS

MR

Title

2525

Fix BDCOM.xPON profile

2733

Add BDCOM.IOS.get_inventory script

2820

BDCOM.IOS.get_mac_address_table. Fix colected interface on mac table.

2820

BDCOM.IOS.get_mac_address_table. Fix colected interface on mac table.

Cisco.IOS

MR

Title

2788

Add mpls l2vp to Cisco profile.

2790

Fix Cisco vLAN subif ifindex matching

2794

Add Cisco IOS SNMP metrics for PPPoE and CPU Load

2883

Add suport for WS-C4900M switch to Cisco.IOS.get_inventory script

2961

Update Cisco.IOS profile

2965

Cisco.IOS. Fix confdb normalizer interface speed error.

2974

Cisco.IOS.get_switchport. Fix vlan_id 0 on SNMP output.

2989

Cisco.IOS.get_cdp_neighbors. Add execute_snmp method.

2995

Cisco.IOS. confdb normalizer fix range notation for vlan.

Dahua.DH

MR

Title

2456

Fix typo in Dahua.DH confdb normalizer.

2524

Add RVi vendor to Dahua.DH profile.

2557

Dahua.DH. Add Web Auth middleware abd getting PTZ version to attributes.

EdgeCore.ES

MR

Title

2392

Fix EdgeCore.ES.get_version script

2466

Resolve “ConfDB: EdgeCore-ES ntp”

2522

Edcore.ES.get_switchport. Fix ‘interface_swport’ regex if output not ‘Native VLAN’ field

2538

confdb edgecore-es. fix ntp address

2691

Fix EdgeCore.ES.get_lldp_neighbors script

2789

EdgeCore.ES.get_lldp_neighbors. Fix port_id local converting bug.

2910

add_snmp_Edgecore.ES_profile

2953

fix_get_portchannel_Edge-Core_profile

Eltex.MES

MR

Title

2394

Add line_wrapper and row_wrapper arguments to parse_table function

2471

Resolve “ConfDB: Eltex-MES ntp”

2484

Bring matchers to Eltex.MES profile

2511

Fix method name on Eltex.MES confdb normalizer.

2569

Fix Eltex.MES.get_inventory script

2597

Fix Eltex.MES.get_capabilities script

2662

Add support for Eltex.MES24xx profile

2701

Fix Eltex.MES.get_lldp_neighbors script

2723

Fix Eltex mes24xx profile

2750

fix_eltex_mes24xx_enable_mode

2781

Add Eltex.MES24xx.get_lldp_neighbors script

2792

Fix Eltex.MES.get_inventory script

2830

Eltex.MES Fix confdb normalizer traceback.

2875

fix_Environment-Temperature_Eltex.MES

2877

Eltex.MES. Clean log message on console after login.

2882

Fix Eltex.MES.get_interfaces script

2988

Eltex.MES.get_lldp_neighbors. Move snmp to generic script.

Extreme.ISW

MR

Title

2873

Extreme.ISW. Add new SA profile.

Extreme.XOS

MR

Title

2668

Fix Extreme.XOS.get_portchannel script

2802

Fix Extreme.XOS.get_interfaces script

2980

Extreme.XOS. Fix detecting stack for X480-24x model.

Generic

MR

Title

2689

Add interface errors delta metrics.

2715

Generic. Add network stp topology changes delta metric.

2767

Add “Network | STP” check to STP Topology metrics.

2783

Add ability to get port name from another table

2865

Generic.get_mpls_vpn. Fix rd param as list.

2904

new_get_inventory_Generic_profile

2919

Add revision into Generic.get_inventory script

2937

fix_get_portchannel_Generic_profile

2968

Generic.get_inventory. Fix KeyError if not ‘HW version’.

2975

Generic.get_capabilities. Add ‘SNMP | OID | EnterpriseID’ caps.

2976

Generic.get_lldp_neighbors.Fix utf-8 decode.

2998

Generic.get_beef improve worked.

Hikvision.DSKV8

MR

Title

2435

Fix Hikvision.DSKV8 channel_name

2465

Fix empty text on Hikvision.DSKV8.get_config.

2523

Hikvision.DSKV8. Fix trace when returned bad xml.

Huawei.MA5600T

MR

Title

2477

Fix mailformed ADSL interface name in Huawei.MA5600T.get_interfaces.

2573

Fix get_metrics MA5600T

2697

Huawei.MA5600T.get_inventory. Add getting daighter card for MA5600.

2704

Huawei.MA5600T add xDSL metrics type.

2706

Huawei.MA5600T.get_inventory. Fix ma5626g chassis.

Huawei.VRP

MR

Title

2482

ConfDB: Allow over-dedent in indent tokenizer

2485

Fix Huawei.VRP matchers for S5628F model.

2517

Fix Huawei.VRP “Are you sure” pattern_more.

2635

Fix STP detection in Huawei.VRP.get_capabilities script

2635

Fix STP detection in Huawei.VRP.get_capabilities script

2777

Huawei.VRP.get_mpls_vpn. Fix shifting string output trace.

2787

Huawei.VRP. Add mpls l2vpn to profile.

2801

Huawei.VRP.get_version. Fix CX600 detect version.

2809

Add Huawei Entity extent mib.

2809

Add Huawei Entity extent mib.

2818

Huawei.VRP.get_inventory. Use display elabel command.

2821

Huawei.VRP.get_inventory. Add interface binding to L2 MPLS.

2831

Huawei.VRP. Fix vlan batch parser on confdb normalizer.

2870

Huawei.VRP.get_lacp_neighbors. Fix trace on bad table.

2971

noc/noc#1082 Huawei.VRP.confdb. Add parse NTP client settings.

Iskratel.MSAN

MR

Title

2431

Add pattern_operation_error to Iskratel.MSAN profile.

2739

Iskratel.MSAN. Add xDSL metrics.

Juniper.JUNOS

MR

Title

2470

Resolve “ConfDB: Juniper-JUNOS ntp”

2659

Fix Juniper.JUNOS.get_inventory script

2672

Fix Juniper.JUNOS.get_inventory script

2681

Add SIB connector type for Juniper EX8200 switches

2711

Add more types in Juniper.JUNOS.get_inventory script

2771

Fix Juniper.JUNOS.get_inventory script

2858

Juniper.JunOS.get_interfaces. Add VPLS,VLL interface mapping.

MikroTik.RouterOS

MR

Title

2454

Resolve “ConfDB: RouterOS ntp”

2645

Update MikroTik.RouterOS.get_inventory.py: add get transceivers info.

2695

Update MikroTik.RouterOS.get_license.py: fix error on CHR platform.

NAG.SNR

MR

Title

2410

Update NAG.SNR.get_portchannel.py

2417

Fix pager in NAG.SNR profile

2462

Update NAG.SNR profile

2540

Add nag.snr.get_inventory

2658

Fix NAG.SNR.get_version script

2702

Update NAG.SNR.get_inventory.py

Qtech.QSW2800

MR

Title

2430

Delete include from command in Qtech.QSW2800 profiles.

2444

Update Qtech.QSW8200 profile

2468

Resolve “ConfDB: Qtech-QSW2800 ntp”

2539

confdb qtech_qsw2800. fix ntp address

2728

Add Qtech.QSW8200.get_inventory script

2815

Qtech.QSW2800.get_inventory. Fix trace when return result without ‘HW version’.

2815

Qtech.QSW2800.get_inventory. Fix trace when return result without ‘HW version’.

2981

Qtech.QSW2800.get_capabilities. Add has_stp_snmp method.

Raisecom.ROS

MR

Title

2476

Add Rotek vendor to Raisecom.ROS profile.

2581

Raisecom.ROS.get_interfaces. Fix getting iface_type.

2866

Raisecom.ROS. Fix detect interface type profile method.

2945

Raisecom.ROS. Fix working for Rotek switches.

2951

noc/noc#1162 Raisecom.ROS. Fix get_lldp_neighbors port regex.

SKS.SKS

MR

Title

2616

Update SKS.SKS profile

2683

Fix prompt at SKS.SKS profile

2812

SKS.SKS.get_interfaces. Fix stuck CLI command when discovery run.

Siklu.EH

MR

Title

2863

fix Siklu get_inventory

Ubiquiti.AirOS

MR

Title

2744

Update Ubiquiti.AirOS profile

Zyxel.MSAN

MR

Title

2743

Fix Zyxel.MSAN profile

2752

Fix Zyxel.MSAN.get_interfaces script

rare

MR

Title

2312

Intracom.Ultralink

2401

Add support ESCOM L to Iskrate.ESCOM profile.

2425

Fix dlink_smart interfaces output

2531

Update Extreme.Summit200 profile

2534

Add new profile TP-Link EAP

2550

Update Angtel.Topaz profile

2578

Update Alstec.MSPU profile

2603

DCN.DCWL.get_interfaces. Filter “00:00:00:00:00:00” MAC.

2619

Update Zhone.MXK profile

2633

fix cisco smb get switchport

2644

Update Upvel.UP profile

2661

Fix Upvel.UP.get_inventory script

2667

Fix vlan parsing in BDCOM.xPON.get_interfaces script

2676

Raisecom.RCIOS add get_inventory.

2694

Eltex.WOP.get_metrics - Disable interface metrics on CLI.

2705

Eltex.DSLAM. Add base xDSL metrics.

2708

Fix Infinet get_lldp_neighbors and get_capabilities

2720

Eltex.WOP.get_metrics. Fix Radio | Tx Power metric.

2731

Fix Eltex.MA4000.get_inventory script

2732

Add Vitesse.VSC.get_inventory script

2737

Fix Vitesse.VSC.get_interfaces script

2738

DLink.DAS. Add xDSL metrics and get_inventory script.

2740

Add base support ECI.HiFOCuS profile.

2746

Add Eltex.RG.get_inventory script

2759

Rotek.RTBSv1 profile SNMP Prefer.

2776

Update Zhone.Bitstorm profile

2803

Upvel.UP. Add get_interface_names to profile for LLDP.

2803

Upvel.UP. Add get_interface_names to profile for LLDP.

2804

Alcatel.7302. Add get_inventory script.

2852

ECI.HiFOCuS. Add get_inventory, Improvement SNMP work.

2853

add Nateks NetXpert get_inventory

2854

add H3C VRP get_inventory

2864

Rotek.RTBS.get_interfaces. Fix traceback when unknown iface type.

2872

Eltex.LTP. Add SNMP support.

2885

Add Intracom docs.

2911

fix_get_inventory1

2920

Fix some get_version script to unify attribures

2927

noc/noc#1159 Add support for MikroTik.SwOS profile

2970

noc/noc#1183 DLink.DAS.get_inventory. Fix serial number regex.

2983

ECI.HiFOCus. SNMP method prefer for working.

2999

ECI.HiFOCus. Prefer use SNMP.

3000

Eltex.LTE. Add operation error.