NOC 19.2

In accordance to our Release Policy we’re proudly present release 19.2.

19.2 release contains of 337 bugfixes, optimisations and improvements.

Highlights

ConfDB

19.2 introduces new configuration processing framework, the ConfDB. ConfDB is the crystallisation of our over decade experience of configuration processing. Set of carefully designed tools, clean API and flexible query language offer the unified way of configuration parsing, validation, processing and converting. Target area covers data extraction (without complex scripts), config validation, querying and inter-vendor configuration conversions.

We built ConfDB around idea of abstract intermediate config representation, somewhat similar with programming languages’ virtual machines, query language with Prolog-like logical programming capabilities, and common pipeline for config parsing.

Key features comparison against existing config processing models:

Area

Old Model

ConfDB

Configuration fetching

Via get_config script

get_config script, external storage download

Configuration storing

GridVCS

GridVCS with compression

Configuration parsing

pyparsing parsers

2-stage, tokenization and normalization

Parser complexity

High

tokenizers are reused between profiles, normalizers - easy

Parser performance

Low

High

Config model

Plain list of predefined facts

tree-like structure

Custom facts

Almost impossible

Easy

Query language syntax

CLIPS

Python

Query debugging

shell

Built-in IDE

Discovery integration

None

Config-driven discovery checks

19.2 contains ConfDB support for EdgeCore.ES, Eltex.MES, Huawei.VRP, Juniper.JUNOS, MikroTik.RouterOS and Qtech.QSW2800 profiles.

Read more …

Headless Mode

ConfDB allows the headless mode of operations, where “headless” means no direct access to equipment. Instead, configuration files are downloaded from external storage via FTP, SFTP or S3 API, stored to GridVCS and processed via ConfDB. Discovery can extract capabilities, interfaces and vlans directly from config. IPAM is also integrated with ConfDB, allowing to track vpns, address and prefixes usage.

Headed/Headless mode may be selected on per-device or per-profile basics via appropriate policies.

Headless mode is a valuable tool for due diligences, security audits and forensics.

Config fetch policy

NOC got additional configuration to select, which configuration to prefer - startup or running. Config fetch policy may be applied on per-profile or per-object level.

IPAM

IPAM got various improvements, including:

  • separate calculation of address and prefix usage

  • prefix profile’s setting to allow reuse of special addresses (network and broadcast)

Ethernet services

IGetInterfaces got the ES address family, allowing to describe complex vlan tag manipulations, often referred as Ethernet Services, Q-in-Q and so on.

GridVCS improvements

19.2 contains simple Mercurial’s mpatch implementation to read old deltas. So NOC is no longer depends on Mercurial.

New profiles

19.2 brings support for various IP-cams, including video-specific metrics. Hikvision.DSKV8, Dahua.DH, Axis.VAPIX profiles support lots of IP-cams out-of-the-box.

In addition, GWD.GFA profile has been added.

ProfileChecker

Profile checking engine (discovery’s profile check) has been refactored to separate class available both for discovery and for custom scripting.

DNS Zone expiration checks

19.2 restores DNS Zone expiration check, regressed long ago in microservices branch.

API Improvements

Script http client

Scripts’ http client has been reworked and got middleware support, allowing various wierd authentication schemes. Integration with web-only managed devices became easier.

nbi telemetry API

NBI service got a telemetry API for metrics pushing from remote agents, making the NOC full-blown telemetry collection and processing system for custom IoT applications.

Code quality

PEP8

Code Quality always has been one of the major priorities. 19.2 contains lots of code cleaning, improving overall code quality and PEP8 compliance.

Tests

Testing infrastructure has been improved and we’d reached 100% test suite passing. Tests passing made the requirement for merge request acceptance.

Python3 compatibility

Python 2.7 will reach end-of-life at 2020-01-01, so we started complex porting project. Additional py3 compatibility linter has been added to the testing lint stage. Python3 compatibility became the requirement for merge request acceptance.

Additional py3 infrastructure for test suite running is ready and we’re working on 100% tests passing.

Breaking Changes

IGetConfig

IGetConfig interface got additional policy input parameter. get_config scripts should recognize it and behave accordingly.

Possible values:

  • r - prefer running config

  • s - prefer startup config

Migration

Migration from 19.2 release is straightforward. Just deploy 19.2 or stable-19.2 tags from Tower.

New features

MR

Title

1568

IGetInterfaces: ethernet services

1582

Config tokenizers

1608

#980 Config normalizers

1647

ConfDB: Database and logic inference engine

1661

Juniper.JUNOS: ConfDB tokenizer

1696

Discovery: Headless config check

1706

nbi: telemetry API

1776

discovery: ConfDB methods

1783

ConfDB: Applicators

1800

Config Fetch Policy

1811

ConfDB: query frontend

1837

discovery: Running policy

1855

Domain expiration check handlers

1872

Bye-bye HG

1894

#38 Add prefix_special_address_policy to Prefix Profile.

1926

ProfileChecker class

Improvements

MR

Title

1505

fm.alarm: Check alarm is not already acknowledged/unacknowledged

1506

Update AuditTrail funcional

1520

Optimize IPv4 to integer conversion

1521

Header panel: add last update button

1542

Add support for ISCOM2608G

1545

Report control initial commit

1550

Patch uncol fm alarm sort

1551

Update Managed Object profile form

1553

Add 40G interfaces support to Cisco.IOS profile

1554

ManagedObjectProfile: Refer to default caps profile by default

1558

Update HP.ProCurve profile

1559

fm-alarm i18n

1562

Add hasAny and hasAll function to BI query language.

1564

Add basic authentication to script.http methods.

1570

Add initial support for GWD.GFA profile

1574

Add pattern_unprivileged_prompt to Iskratel.MSAN.

1575

Add http/https proto to script command.

1583

Set enable datasource default settings to False.

1587

pytest 4.1.1

1588

HTTP Client auto JSON requests

1585

Update GWD.GFA profile

1592

Move FM reportalarmdetail application to ControlReport.

1594

Add Interface Errors BIP metric to Huawei.MA5600T.

1596

Script: http sessions, headers and cookies control

1609

Add interval to PM scale function.

1611

Add Hikvision.DSKV8 profile

1612

Move isInMaintenance field on fm.alarm to bulk.

1613

fm.alarm: alarm_class multi-select

1630

combotree: reset pageSize limit

1634

#980 noc.core.confdb package

1645

Digest middleware

1655

Update SKS.SKS profile

1648

Huawei ma5600t

1660

parse_table: Mixed tabs/space processing

1662

NetworkSegment.get_nested_ids: Rewrite to fix out-of-memory condition

1663

Update BDCOM.xPON profile

1664

Add more d-link switches support

1666

i18n: inv.reportifacestatus

1667

Add more models to Eltex.MES profile

1669

Add Dahua.DH profile for IP cam.

1671

Add “port id subtype” to Huawei.VRP.get_lldp_neighbors.

1680

header panel: hide last update, when don’t use

1681

Add initial support for ZTE.ZXA10 profile

1684

BEREncoder/BERDecoder singleton

1687

BER: Opaque/float/double type decoders

1691

Add patch attribute to Huawei.VRP.get_version.

1692

Add CFP TransEth40G in verify-model.py for Finistar FTLQ8181EBLM

1693

remote sync date

1694

Add base implementation TCP RTSP client.

1695

Catch RPC Error on STP discovery.

1698

Add capabilities to script matchers.

1705

Add BRAS capabilities to Juniper.JUNOS profile

1711

ClickHouse client: Allow empty passwords

1716

Add group by slot to `Memory

1721

MikroTik.RouterOS: Match multi-line prompts

1733

Add matchers to Alstec.24xx.get_metrics.

1737

Update ZTE.ZXA10 profile

1738

discovery: Split config fetching and config validation stages

1742

Add state decorator to ip.vrf API.

1743

Add ack_ts, ack_user field to Alarms BI model

1750

Add serials number field to managed_object BI models and extractor.

1753

Check handler permission before call.

1758

Add ManagedObjectAttribute to card template.

1761

Add initial support for Axis.VAPIX profile

1762

Add Dahua.DH.get_local_version script.

1763

Add Beward.BD.get_local_users script.

1767

ConfDB: Customized syntax

1774

Profile’s confdb package

1775

Update Ericsson SEOS profile

1779

Migrate NBI loader to BaseLoader.

1781

Normalizer: defer() and RouterOS normalizer skeleton

1784

Add datastream meta field to ensure-indexes command.

1791

Update Juniper.JUNOS.get_interfaces script to support tunnels

1794

Add operation error in Huawei.MA5600T profile

1796

Do not expand tabs in parse_table() on some rotten devices

1814

ConfDB: VRF configuration

1820

Eltex.MES: MES-2324B/MES-2348B platforms

1822

ConfDB: RebaseApplicator

1833

Update get_portchannel.py

1835

Show service’s startup time

1836

discovery: Load ConfDB on demand even if config check is disabled

1858

Nocmc integration

1859

add lldp zyxel interface

1860

External Storage: Check directory is not exists before creation

1865

Add Jiniper.JunOS confdb normalizer.

1868

ConfDB tests

1873

Docker CODEOWNERS

1877

Split docker-compose projects

1878

Profile: Show DeprecationWarning when profile is not in profile.py

1879

Optimize custom field loading

1881

pymongo 3.8, mongoengine 0.17.0

1883

Add mongodb rs init info. Pretty hacky

1884

Add future 2 stage. Make it mandatory

1887

Add Address Usage column to IPAM.

1896

VC.VLAN: add Description column, add interface name to L3 interfaces list

1907

Add future to docker requirements.

1910

test: Report empty module’s docstrings

1911

#996 Add project field to Subscriber and Supplier.

1913

Exclude self when AddressRange overlaps check when save.

1914

Getting vrf_id when confdb interface discovery.

1916

ConfDB: vpn-id

1919

Add type field to Qtech.QSW2800 memory_usage metric.

1921

Deny move cpestatus to anothet managed_object when it inactive

1922

Clean etl mapping key when load.

1923

Qtech.QSW2800 more more patterns

1924

Replace generic profile getting method in report_objectstat.

1925

Reorganize model tests

1927

Add support for SWA-2200-16P-AC device

1935

Add Patch Version column and autowidth option to Object Detail Repport.

1936

Remove volumes also when down tests

1949

Add export_beef parameter to script.

1955

Add Eltex.MES confdb normalizer.

1959

Move MongoCache index creation into ensure-indexes

1960

Replace ensure_index with create_index

1964

.to_json() and .get_json_path() tests

1978

ipam: Policy defaults in “New Prefix” form

1992

Add SNMP method to Huawei.MA5600T.get_metrics.

1995

#1030 Add DataSourceCache to ensure-index.

Bugfixes

MR

Title

1495

Fix Rotek.RTBSv1 cli errors

1496

Fix Rotek.RTBS cli errors

1504

Fix interfaceattributes datasource filter.

1509

Fix Qtech.QSW.get_switchport script

1511

Fix match trace on SKS.SKS profile.

1513

Fix MACDB circular references.

1517

Fix is_builtin field

1519

Fix threshold profile label in managed object profile metrics view.

1524

Fix Eltex.MES Lldp by snmp

1526

#998 Send bi_id field to frontend as string.

1528

Fix noc script command. Clean update-spec collect command.

1530

Fix TFortis.get_version script.

1531

Revert DLink.DxS.get_spanning_tree changes.

1533

Fix Alcatel.TIMOS.get_interfaces empty mac.

1536

Fix translation command BI model path

1540

Fm alarm fix

1541

localization fix (min)

1543

Fix Extreme.XOS.get_lldp_neighbors script

1544

Fix Detail Alarm report

1546

Fix convert interface name for Qtech.QSW

1547

Fix Juniper.JUNOS.get_mac_address_table script

1548

Fix clean thresholdprofile migration

1549

Fix get_lldp_neighbors script for Juniper.Junos

1552

Fix Cisco.ASA parse config

1555

Fix Time Pattern

1557

Cisco.SMB

1566

Fix DCN.DCWL.get_interfaces typo.

1569

fm-alarm save filters in local store

1571

Fix Qtech.QSW profile for lldp

1576

Fix Control translation.

1577

Fix Siklu.EH.get_lldp_neighbors.

1586

Fix Alcatel.AOS.get_switchport script

1597

#1006 Fix wipe user

1598

Fix detect snmp v1 for profile check rule.

1601

Fix error in reporting ifSpeed on some D-Link devices

1603

Fix syslog CH Model message field description.

1610

#1009 Fix Add/Insert object in inventory tree.

1614

fm.alarm: view recent alarms fixed

1615

Fix Raisecom.ROS.get_interfaces trace.

1617

Add metrics to selfmon fm collector.

1619

Fix Huawei.MA5600T.get_interfaces get vlans.

1623

Fix Raisecom.ROS.get_interfaces script

1624

Fix DLink.DxS_Smart.get_lldp_neighbors script

1625

Fix Eltex.MES5448.get_version script

1626

Fix Interfaces status ex for physical interfaces

1627

use ExtJS .endsWith for IE compatibility

1632

Some fixes for Huawei.MA5600T profile.

1636

Fix import

1637

beward_profile

1638

Fix Eltex.MES.get_spanning_tree script

1639

Fix for Oob and Stack-port interfaces

1641

#990 grid.column, override formatter to ‘htmlEncode’

1642

Fix Eltex.MES.get_interfaces script

1643

Fix Cisco.IOSXR.get_inventory script

1644

Fix discovery cpe

1649

Change replace character for Grafana PM Dashboard

1652

Add fix_cpe_header function to Huawei.MA5600T profile.

1654

grid.column, override formatter fix

1656

Fix DLink.DxS_Industrial_CLI interface naming

1657

Fix DLink.DxS.get_mac_address_table script

1658

sova/rtk#671

1659

#1018

1670

Fix snmp_metrics_get_chunk on Rotek.BT profile.

1677

Fix stupid copy/paste error

1683

Config MUST NOT match re validation policy

1689

Fix DCN.DCWL.get_metrics scale and type Interface metrics.

1697

fix juniper.junos metric Subscribers (for version >16) and Temperature

1700

Fix Raisecom.ROS.get_inventory trace when Unknown SFP.

1702

Fix Siklu.EH.get_lldp_neighbors empty section trace.

1712

Fix Rotek RTBSv1 profile

1718

Fix CPU, Memory, Temperature metrics for Huawei.VRP

1722

Some fixes on IP Cam profiles Dahua.DH, Beward.BD, Hikvision.DSKV8

1723

juniper.junos metric Subscribers without caps

1726

Huawei.MA5600T move profile settings to profile.py

1728

juniper.junos metric Subscribers without caps

1731

Alstec.24xx.get_metrics fix trace when no controller module.

1734

Fix matchers when check string in iter.

1735

Fix clickhouse backend IPv4 displayed field.

1741

#977 Fix collect SLA metrics in Cisco.IOS.get_metrics.

1745

Fix Huawei.MA5600T.get_interfaces script

1747

#939 Fix Huawei.VPR profile system patter_prompt

1749

Fix chwriter channels_active metric trace.

1751

#973 Fix managed_object two in card path exception.

1752

Fix CursorTimeout when execute datastream rebuild command on large collection.

1754

Fix previous commit

1755

Fix Cisco.IOS.get_interfaces script

1756

Fix Orion.NOS profile

1757

Fix Juniper.JUNOS.get_version script

1760

Fix fm models. add json_unique_fields

1780

Fix exception in DLink.DxS_Smart profile

1788

Fix metrics in card

1789

Fix MikroTik.RouterOS.get_interfaces script

1793

Fix low privelegies error in DLink.DxS profile

1799

fix

1802

Fix box.job typo.

1805

Fix Cisco.NXOS.get_inventory script

1817

Fix DLink.DxS clean_lldp_neighbor() function

1838

Fix Eltex.MES.get_lldp_neighbors script

1839

Fix MikroTik.RouterOS.get_lldp_neighbors script

1840

Fix Eltex.RG.get_version script

1841

Fix Eltex.DSLAM profile

1842

Fix Hiawei.MA5300. Disable session use.

1844

Fix Qtech.QSW2500 script

1852

Fix ManagedObject card not shows interfaces without metrics.

1857

#1014 Fix ‘Config MUST NOT match string’ validator

1861

Fix row_limit in web app.

1864

Fix ch-policy dot name on command.

1867

Fix ManagedObject vendor migration.

1869

Fix wrong scripts name

1871

Fix tests

1874

py3: Fix threadpool

1875

fix get_config tplink t2600 confdb

1876

Fix tests

1889

Fix MikroTik.RouterOS profile

1890

Fix SKS.SKS.get_spanning_tree script

1897

fix get_inv for X650

1899

Fix to_python method for Clickhouse nested field.

1901

Fix SKS.SKS convert_interface_name() function

1902

Fix SKS.SKS.get_spanning_tree script

1903

Fix Iskratel.ESCOM.get_spanning_tree script

1904

fix_metric_memory

1905

Fix Nateks.FlexGain.get_interfaces script

1908

Fix limit check on extapplication.

1928

Fix DLink.DxS.get_interface_status_ex script

1929

Revert “Merge branch ‘qsw2800-more-patterns’ into ‘master’”

1930

Fix Alstec.24xx.get_interfaces script

1931

Catch SNMP Error when Huawei.VRP SNMP worked.

1938

Fix classifier rule cleanup code.

1939

Some fixes for Hikvision.DSKV8 profile.

1940

Fix get_handler error message typo.

1942

Fix CPEStatus discovery when calculate diff for new cpe.

1943

Fix Huawei.MA5600T pattern more for Honet.

1946

Fix multiple jobs on master branch conflict. Prettify a little bit.

1950

Fix Huawei.VRP.get_capabilities snmp check when get.

1954

Some fixes in Generic snmp scripts.

1961

Fix choices attribute in StringList interface parameter.

1966

Fix Juniper.JunOS confdb normalizer.

1967

Fix job_handler validator in Workflow editor backend.

1979

Fix Cisco.IOS.get_version script

1986

LineTokenizer: Fix string_quote processing

1987

Fix Qtech.QSW2800.get_inventory datetime import.

1989

Fix help command execute on custom.

1991

Fix get_config script on profiles for policy.

1993

#1041 Fix segment update_summary fix.

1994

#1039 Make IPAM VRF search ungrouped VRF.

1996

Fix calculate address and prefix usage.

2001

Fix reportdatasource backend.

2003

#1044 Fix import error on reportoverlappedoids.

2004

#981 Fix Cisco.ASA pattern_username match to Last login message.

Code cleanup

MR

Title

1591

Remove wildcard imports

1604

DocInline code cleanup

1605

Remove inv and fm monitor deprecated API

1606

Remove cocomo report

1771

fix_YAMLLoad_safe_load

1792

Set yapf column limit to 120

1797

flake8: Disable W605

1808

cm_migrations. code refactoring

1809

crm_migrations. code refactoring

1810

dns_migrations. code refactoring

1812

fm_migrations. code refactoring

1813

gis_migrations. code refactoring

1815

inv_migrations. code refactoring.

1818

ip_migrations. code refactoring

1819

kb_migrations. code refactoring

1821

main_migrations. code refactoring

1823

peer_migrations. code refactoring

1824

phone_migrations. code refactoring

1825

pm_migrations. code refactoring

1826

project_migrations. code refactoring

1828

sa_migrations. code refactoring

1829

sla_migrations. code refactoring

1830

vc_migrations. code refactoring

1831

wf_migrations. code refactoring.

1832

Update init.py

1850

flake8: Fix tests/

1851

Remove lib/tt.py

1856

Remove obsolete periodics

1870

Cleanup noc.settings import.

1880

py3: Rename izip_longest

1882

Cleanup init.py

1891

flake8_E2_E3_ver2 without sa_profiles, services, tests

1892

fix_migrations. DeprecationWarning pymongo

1898

flake8_E2_E3_ver2. services

1909

flake8_E2_E3_ver2. sa_profiles

1915

Fix @on_delete_check decorators

1918

Code cleanup

1958

Cleanup DeprecationWarnings

1962

Remove socketKeepAlive option

1968

flake8_f401_webapp_cm

1969

flake8_f401_webapp_crm

1970

flake8_f401_webapp_dns

1971

flake8_f401_webapp_fm

1972

flake8_f401_webapp_gis

1973

flake8_f401_webapp_inv

1974

flake8_f401_webapp_main

1975

flake8_f401_webapp_maintenance

1976

lib.app cleanup

1980

2to3 idioms

1981

del_with_statement

1982

2to3 basestring

1983

2to3 set literal

1984

Port legacy tests

1985

2to3 metaclass