Skip to content

NOC 19.2

In accordance to our Release Policy <releases-policy> we're proudly present release 19.2.

19.2 release contains of 337 bugfixes, optimisations and improvements.

Highlights

ConfDB

19.2 introduces new configuration processing framework, the ConfDB. ConfDB is the crystallisation of our over decade experience of configuration processing. Set of carefully designed tools, clean API and flexible query language offer the unified way of configuration parsing, validation, processing and converting. Target area covers data extraction (without complex scripts), config validation, querying and inter-vendor configuration conversions.

We built ConfDB around idea of abstract intermediate config representation, somewhat similar with programming languages’ virtual machines, query language with Prolog-like logical programming capabilities, and common pipeline for config parsing.

Key features comparison against existing config processing models:

Area Old Model ConfDB
Configuration fetching Via get_config script get_config script, external storage download
Configuration storing GridVCS GridVCS with compression
Configuration parsing pyparsing parsers 2-stage, tokenization and normalization
Parser complexity High tokenizers are reused between profiles, normalizers - easy
Parser performance Low High
Config model Plain list of predefined facts tree-like structure
Custom facts Almost impossible Easy
Query language syntax CLIPS Python
Query debugging shell Built-in IDE
Discovery integration None Config-driven discovery checks

19.2 contains ConfDB support for EdgeCore.ES, Eltex.MES, Huawei.VRP, Juniper.JUNOS, MikroTik.RouterOS and Qtech.QSW2800 profiles.

Read more

Headless Mode

ConfDB allows the headless mode of operations, where “headless” means no direct access to equipment. Instead, configuration files are downloaded from external storage via FTP, SFTP or S3 API, stored to GridVCS and processed via ConfDB. Discovery can extract capabilities, interfaces and vlans directly from config. IPAM is also integrated with ConfDB, allowing to track vpns, address and prefixes usage.

Headed/Headless mode may be selected on per-device or per-profile basics via appropriate policies.

Headless mode is a valuable tool for due diligencies, security audits and forensics.

Config fetch policy

NOC got additional configuration to select, which configuration to prefer - startup or running. Config fetch policy may be applied on per-profile or per-object level.

IPAM

IPAM got various improvements, including:

  • separate calculation of address and prefix usage
  • prefix profile’s setting to allow reuse of special addresses (network and broadcast)

Ethernet services

IGetInterfaces got the ES address family, allowing to describe complex vlan tag manipulations, often referred as Ethernet Services, Q-in-Q and so on.

GridVCS improvements

19.2 contains simple Mercurial’s mpatch implementation to read old deltas. So NOC is no longer depends on Mercurial.

New profiles

19.2 brings support for various IP-cams, including video-specific metrics. Hikvision.DSKV8, Dahua.DH, Axis.VAPIX profiles support lots of IP-cams out-of-the-box.

In addition, GWD.GFA profile has been added.

ProfileChecker

Profile checking engine (discovery’s profile check) has been refactored to separate class available both for discovery and for custom scripting.

DNS Zone expiration checks

19.2 restores DNS Zone expiration check, regressed long ago in microservices branch.

API Improvements

Script http client

Scripts’ http client has been reworked and got middleware support, allowing various wierd authentication schemes. Integration with web-only managed devices became easier.

nbi telemetry API

NBI service got a telemetry API <api-nbi-telemetry> for metrics pushing from remote agents, making the NOC full-blown telemetry collection and processing system for custom IoT applications.

Code quality

PEP8

Code Quality always has been one of the major priorities. 19.2 contains lots of code cleaning, improving overall code quality and PEP8 compliance.

Tests

Testing infrastructure has been improved and we’d reached 100% test suite passing. Tests passing made the requirement for merge request acceptance.

Python3 compatibility

Python 2.7 will reach end-of-life at 2020-01-01, so we started complex porting project. Additional py3 compatibility linter has been added to the testing lint stage. Python3 compatibility became the requirement for merge request acceptance.

Additional py3 infrastructure for test suite running is ready and we’re working on 100% tests passing.

Breaking Changes

IGetConfig

IGetConfig interface got additional policy input parameter. get_config scripts should recognize it and behave accordingly.

Possible values:

  • r - prefer running config
  • s - prefer startup config

Migration

Migration from 19.2 release is straightforward. Just deploy 19.2 or stable-19.2 tags from Tower.

New features

MR Title
MR1568 IGetInterfaces: ethernet services
MR1582 Config tokenizers
MR1608 #980 Config normalizers
MR1647 ConfDB: Database and logic inference engine
MR1661 Juniper.JUNOS: ConfDB tokenizer
MR1696 Discovery: Headless config check
MR1706 nbi: telemetry API
MR1776 discovery: ConfDB methods
MR1783 ConfDB: Applicators
MR1800 Config Fetch Policy
MR1811 ConfDB: query frontend
MR1837 discovery: Running policy
MR1855 Domain expiration check handlers
MR1872 Bye-bye HG
MR1894 #38 Add prefix_special_address_policy to Prefix Profile.
MR1926 ProfileChecker class

Improvements

MR Title
MR1505 fm.alarm: Check alarm is not already acknowledged/unacknowledged
MR1506 Update AuditTrail funcional
MR1520 Optimize IPv4 to integer conversion
MR1521 Header panel: add last update button
MR1542 Add support for ISCOM2608G
MR1545 Report control initial commit
MR1550 Patch uncol fm alarm sort
MR1551 Update Managed Object profile form
MR1553 Add 40G interfaces support to Cisco.IOS profile
MR1554 ManagedObjectProfile: Refer to default caps profile by default
MR1558 Update HP.ProCurve profile
MR1559 fm-alarm i18n
MR1562 Add hasAny and hasAll function to BI query language.
MR1564 Add basic authentication to script.http methods.
MR1570 Add initial support for GWD.GFA profile
MR1574 Add pattern_unprivileged_prompt to Iskratel.MSAN.
MR1575 Add http/https proto to script command.
MR1583 Set enable datasource default settings to False.
MR1587 pytest 4.1.1
MR1588 HTTP Client auto JSON requests
MR1585 Update GWD.GFA profile
MR1592 Move FM reportalarmdetail application to ControlReport.
MR1594 Add Interface Errors BIP metric to Huawei.MA5600T.
MR1596 Script: http sessions, headers and cookies control
MR1609 Add interval to PM scale function.
MR1611 Add Hikvision.DSKV8 profile
MR1612 Move isInMaintenance field on fm.alarm to bulk.
MR1613 fm.alarm: alarm_class multi-select
MR1630 combotree: reset pageSize limit
MR1634 #980 noc.core.confdb package
MR1645 Digest middleware
MR1655 Update SKS.SKS profile
MR1648 Huawei ma5600t
MR1660 parse_table: Mixed tabs/space processing
MR1662 NetworkSegment.get_nested_ids: Rewrite to fix out-of-memory condition
MR1663 Update BDCOM.xPON profile
MR1664 Add more d-link switches support
MR1666 i18n: inv.reportifacestatus
MR1667 Add more models to Eltex.MES profile
MR1669 Add Dahua.DH profile for IP cam.
MR1671 Add “port id subtype” to Huawei.VRP.get_lldp_neighbors.
MR1680 header panel: hide last update, when don’t use
MR1681 Add initial support for ZTE.ZXA10 profile
MR1684 BEREncoder/BERDecoder singleton
MR1687 BER: Opaque/float/double type decoders
MR1691 Add patch attribute to Huawei.VRP.get_version.
MR1692 Add CFP TransEth40G in verify-model.py for Finistar FTLQ8181EBLM
MR1693 remote sync date
MR1694 Add base implementation TCP RTSP client.
MR1695 Catch RPC Error on STP discovery.
MR1698 Add capabilities to script matchers.
MR1705 Add BRAS capabilities to Juniper.JUNOS profile
MR1711 ClickHouse client: Allow empty passwords
MR1716 Add group by slot to `Memory
MR1721 MikroTik.RouterOS: Match multi-line prompts
MR1733 Add matchers to Alstec.24xx.get_metrics.
MR1737 Update ZTE.ZXA10 profile
MR1738 discovery: Split config fetching and config validation stages
MR1742 Add state decorator to ip.vrf API.
MR1743 Add ack_ts, ack_user field to Alarms BI model
MR1750 Add serials number field to managed_object BI models and extractor.
MR1753 Check handler permission before call.
MR1758 Add ManagedObjectAttribute to card template.
MR1761 Add initial support for Axis.VAPIX profile
MR1762 Add Dahua.DH.get_local_version script.
MR1763 Add Beward.BD.get_local_users script.
MR1767 ConfDB: Customized syntax
MR1774 Profile’s confdb package
MR1775 Update Ericsson SEOS profile
MR1779 Migrate NBI loader to BaseLoader.
MR1781 Normalizer: defer() and RouterOS normalizer skeleton
MR1784 Add datastream meta field to ensure-indexes command.
MR1791 Update Juniper.JUNOS.get_interfaces script to support tunnels
MR1794 Add operation error in Huawei.MA5600T profile
MR1796 Do not expand tabs in parse_table() on some rotten devices
MR1814 ConfDB: VRF configuration
MR1820 Eltex.MES: MES-2324B/MES-2348B platforms
MR1822 ConfDB: RebaseApplicator
MR1833 Update get_portchannel.py
MR1835 Show service’s startup time
MR1836 discovery: Load ConfDB on demand even if config check is disabled
MR1858 Nocmc integration
MR1859 add lldp zyxel interface
MR1860 External Storage: Check directory is not exists before creation
MR1865 Add Jiniper.JunOS confdb normalizer.
MR1868 ConfDB tests
MR1873 Docker CODEOWNERS
MR1877 Split docker-compose projects
MR1878 Profile: Show DeprecationWarning when profile is not in profile.py
MR1879 Optimize custom field loading
MR1881 pymongo 3.8, mongoengine 0.17.0
MR1883 Add mongodb rs init info. Pretty hacky
MR1884 Add future 2 stage. Make it mandatory
MR1887 Add Address Usage column to IPAM.
MR1896 VC.VLAN: add Description column, add interface name to L3 interfaces list
MR1907 Add future to docker requirements.
MR1910 test: Report empty module’s docstrings
MR1911 #996 Add project field to Subscriber and Supplier.
MR1913 Exclude self when AddressRange overlaps check when save.
MR1914 Getting vrf_id when confdb interface discovery.
MR1916 ConfDB: vpn-id
MR1919 Add type field to Qtech.QSW2800 memory_usage metric.
MR1921 Deny move cpestatus to anothet managed_object when it inactive
MR1922 Clean etl mapping key when load.
MR1923 Qtech.QSW2800 more more patterns
MR1924 Replace generic profile getting method in report_objectstat.
MR1925 Reorganize model tests
MR1927 Add support for SWA-2200-16P-AC device
MR1935 Add Patch Version column and autowidth option to Object Detail Repport.
MR1936 Remove volumes also when down tests
MR1949 Add export_beef parameter to script.
MR1955 Add Eltex.MES confdb normalizer.
MR1959 Move MongoCache index creation into ensure-indexes
MR1960 Replace ensure_index with create_index
MR1964 .to_json() and .get_json_path() tests
MR1978 ipam: Policy defaults in “New Prefix” form
MR1992 Add SNMP method to Huawei.MA5600T.get_metrics.
MR1995 #1030 Add DataSourceCache to ensure-index.

Bugfixes

MR Title
MR1495 Fix Rotek.RTBSv1 cli errors
MR1496 Fix Rotek.RTBS cli errors
MR1504 Fix interfaceattributes datasource filter.
MR1509 Fix Qtech.QSW.get_switchport script
MR1511 Fix match trace on SKS.SKS profile.
MR1513 Fix MACDB circular references.
MR1517 Fix is_builtin field
MR1519 Fix threshold profile label in managed object profile metrics view.
MR1524 Fix Eltex.MES Lldp by snmp
MR1526 #998 Send bi_id field to frontend as string.
MR1528 Fix noc script command. Clean update-spec collect command.
MR1530 Fix TFortis.get_version script.
MR1531 Revert DLink.DxS.get_spanning_tree changes.
MR1533 Fix Alcatel.TIMOS.get_interfaces empty mac.
MR1536 Fix translation command BI model path
MR1540 Fm alarm fix
MR1541 localization fix (min)
MR1543 Fix Extreme.XOS.get_lldp_neighbors script
MR1544 Fix Detail Alarm report
MR1546 Fix convert interface name for Qtech.QSW
MR1547 Fix Juniper.JUNOS.get_mac_address_table script
MR1548 Fix clean thresholdprofile migration
MR1549 Fix get_lldp_neighbors script for Juniper.Junos
MR1552 Fix Cisco.ASA parse config
MR1555 Fix Time Pattern
MR1557 Cisco.SMB
MR1566 Fix DCN.DCWL.get_interfaces typo.
MR1569 fm-alarm save filters in local store
MR1571 Fix Qtech.QSW profile for lldp
MR1576 Fix Control translation.
MR1577 Fix Siklu.EH.get_lldp_neighbors.
MR1586 Fix Alcatel.AOS.get_switchport script
MR1597 #1006 Fix wipe user
MR1598 Fix detect snmp v1 for profile check rule.
MR1601 Fix error in reporting ifSpeed on some D-Link devices
MR1603 Fix syslog CH Model message field description.
MR1610 #1009 Fix Add/Insert object in inventory tree.
MR1614 fm.alarm: view recent alarms fixed
MR1615 Fix Raisecom.ROS.get_interfaces trace.
MR1617 Add metrics to selfmon fm collector.
MR1619 Fix Huawei.MA5600T.get_interfaces get vlans.
MR1623 Fix Raisecom.ROS.get_interfaces script
MR1624 Fix DLink.DxS_Smart.get_lldp_neighbors script
MR1625 Fix Eltex.MES5448.get_version script
MR1626 Fix Interfaces status ex for physical interfaces
MR1627 use ExtJS .endsWith for IE compatibility
MR1632 Some fixes for Huawei.MA5600T profile.
MR1636 Fix import
MR1637 beward_profile
MR1638 Fix Eltex.MES.get_spanning_tree script
MR1639 Fix for Oob and Stack-port interfaces
MR1641 #990 grid.column, override formatter to ‘htmlEncode’
MR1642 Fix Eltex.MES.get_interfaces script
MR1643 Fix Cisco.IOSXR.get_inventory script
MR1644 Fix discovery cpe
MR1649 Change replace character for Grafana PM Dashboard
MR1652 Add fix_cpe_header function to Huawei.MA5600T profile.
MR1654 grid.column, override formatter fix
MR1656 Fix DLink.DxS_Industrial_CLI interface naming
MR1657 Fix DLink.DxS.get_mac_address_table script
MR1658 sova/rtk#671
MR1659 #1018
MR1670 Fix snmp_metrics_get_chunk on Rotek.BT profile.
MR1677 Fix stupid copy/paste error
MR1683 Config MUST NOT match re validation policy
MR1689 Fix DCN.DCWL.get_metrics scale and type Interface metrics.
MR1697 fix juniper.junos metric Subscribers (for version >16) and Temperature
MR1700 Fix Raisecom.ROS.get_inventory trace when Unknown SFP.
MR1702 Fix Siklu.EH.get_lldp_neighbors empty section trace.
MR1712 Fix Rotek RTBSv1 profile
MR1718 Fix CPU, Memory, Temperature metrics for Huawei.VRP
MR1722 Some fixes on IP Cam profiles Dahua.DH, Beward.BD, Hikvision.DSKV8
MR1723 juniper.junos metric Subscribers without caps
MR1726 Huawei.MA5600T move profile settings to profile.py
MR1728 juniper.junos metric Subscribers without caps
MR1731 Alstec.24xx.get_metrics fix trace when no controller module.
MR1734 Fix matchers when check string in iter.
MR1735 Fix clickhouse backend IPv4 displayed field.
MR1741 #977 Fix collect SLA metrics in Cisco.IOS.get_metrics.
MR1745 Fix Huawei.MA5600T.get_interfaces script
MR1747 #939 Fix Huawei.VPR profile system patter_prompt
MR1749 Fix chwriter channels_active metric trace.
MR1751 #973 Fix managed_object two in card path exception.
MR1752 Fix CursorTimeout when execute datastream rebuild command on large collection.
MR1754 Fix previous commit
MR1755 Fix Cisco.IOS.get_interfaces script
MR1756 Fix Orion.NOS profile
MR1757 Fix Juniper.JUNOS.get_version script
MR1760 Fix fm models. add json_unique_fields
MR1780 Fix exception in DLink.DxS_Smart profile
MR1788 Fix metrics in card
MR1789 Fix MikroTik.RouterOS.get_interfaces script
MR1793 Fix low privelegies error in DLink.DxS profile
MR1799 fix
MR1802 Fix box.job typo.
MR1805 Fix Cisco.NXOS.get_inventory script
MR1817 Fix DLink.DxS clean_lldp_neighbor() function
MR1838 Fix Eltex.MES.get_lldp_neighbors script
MR1839 Fix MikroTik.RouterOS.get_lldp_neighbors script
MR1840 Fix Eltex.RG.get_version script
MR1841 Fix Eltex.DSLAM profile
MR1842 Fix Hiawei.MA5300. Disable session use.
MR1844 Fix Qtech.QSW2500 script
MR1852 Fix ManagedObject card not shows interfaces without metrics.
MR1857 #1014 Fix ‘Config MUST NOT match string’ validator
MR1861 Fix row_limit in web app.
MR1864 Fix ch-policy dot name on command.
MR1867 Fix ManagedObject vendor migration.
MR1869 Fix wrong scripts name
MR1871 Fix tests
MR1874 py3: Fix threadpool
MR1875 fix get_config tplink t2600 confdb
MR1876 Fix tests
MR1889 Fix MikroTik.RouterOS profile
MR1890 Fix SKS.SKS.get_spanning_tree script
MR1897 fix get_inv for X650
MR1899 Fix to_python method for Clickhouse nested field.
MR1901 Fix SKS.SKS convert_interface_name() function
MR1902 Fix SKS.SKS.get_spanning_tree script
MR1903 Fix Iskratel.ESCOM.get_spanning_tree script
MR1904 fix_metric_memory
MR1905 Fix Nateks.FlexGain.get_interfaces script
MR1908 Fix limit check on extapplication.
MR1928 Fix DLink.DxS.get_interface_status_ex script
MR1929 Revert “Merge branch ‘qsw2800-more-patterns’ into ‘master’”
MR1930 Fix Alstec.24xx.get_interfaces script
MR1931 Catch SNMP Error when Huawei.VRP SNMP worked.
MR1938 Fix classifier rule cleanup code.
MR1939 Some fixes for Hikvision.DSKV8 profile.
MR1940 Fix get_handler error message typo.
MR1942 Fix CPEStatus discovery when calculate diff for new cpe.
MR1943 Fix Huawei.MA5600T pattern more for Honet.
MR1946 Fix multiple jobs on master branch conflict. Prettify a little bit.
MR1950 Fix Huawei.VRP.get_capabilities snmp check when get.
MR1954 Some fixes in Generic snmp scripts.
MR1961 Fix choices attribute in StringList interface parameter.
MR1966 Fix Juniper.JunOS confdb normalizer.
MR1967 Fix job_handler validator in Workflow editor backend.
MR1979 Fix Cisco.IOS.get_version script
MR1986 LineTokenizer: Fix string_quote processing
MR1987 Fix Qtech.QSW2800.get_inventory datetime import.
MR1989 Fix help command execute on custom.
MR1991 Fix get_config script on profiles for policy.
MR1993 #1041 Fix segment update_summary fix.
MR1994 #1039 Make IPAM VRF search ungrouped VRF.
MR1996 Fix calculate address and prefix usage.
MR2001 Fix reportdatasource backend.
MR2003 #1044 Fix import error on reportoverlappedoids.
MR2004 #981 Fix Cisco.ASA pattern_username match to Last login message.

Code cleanup

MR Title
MR1591 Remove wildcard imports
MR1604 DocInline code cleanup
MR1605 Remove inv and fm monitor deprecated API
MR1606 Remove cocomo report
MR1771 fix_YAMLLoad_safe_load
MR1792 Set yapf column limit to 120
MR1797 flake8: Disable W605
MR1808 cm_migrations. code refactoring
MR1809 crm_migrations. code refactoring
MR1810 dns_migrations. code refactoring
MR1812 fm_migrations. code refactoring
MR1813 gis_migrations. code refactoring
MR1815 inv_migrations. code refactoring.
MR1818 ip_migrations. code refactoring
MR1819 kb_migrations. code refactoring
MR1821 main_migrations. code refactoring
MR1823 peer_migrations. code refactoring
MR1824 phone_migrations. code refactoring
MR1825 pm_migrations. code refactoring
MR1826 project_migrations. code refactoring
MR1828 sa_migrations. code refactoring
MR1829 sla_migrations. code refactoring
MR1830 vc_migrations. code refactoring
MR1831 wf_migrations. code refactoring.
MR1832 Update init.py
MR1850 flake8: Fix tests/
MR1851 Remove lib/tt.py
MR1856 Remove obsolete periodics
MR1870 Cleanup noc.settings import.
MR1880 py3: Rename izip_longest
MR1882 Cleanup init.py
MR1891 flake8_E2_E3_ver2 without sa_profiles, services, tests
MR1892 fix_migrations. DeprecationWarning pymongo
MR1898 flake8_E2_E3_ver2. services
MR1909 flake8_E2_E3_ver2. sa_profiles
MR1915 Fix @on_delete_check decorators
MR1918 Code cleanup
MR1958 Cleanup DeprecationWarnings
MR1962 Remove socketKeepAlive option
MR1968 flake8_f401_webapp_cm
MR1969 flake8_f401_webapp_crm
MR1970 flake8_f401_webapp_dns
MR1971 flake8_f401_webapp_fm
MR1972 flake8_f401_webapp_gis
MR1973 flake8_f401_webapp_inv
MR1974 flake8_f401_webapp_main
MR1975 flake8_f401_webapp_maintenance
MR1976 lib.app cleanup
MR1980 2to3 idioms
MR1981 del_with_statement
MR1982 2to3 basestring
MR1983 2to3 set literal
MR1984 Port legacy tests
MR1985 2to3 metaclass