Download here - Fognet Consulting
Transcript
Fognet’s Field Guide to OpenView Network Node Manager 2nd Edition Mike Peckar Fogbooks® West Boylston, Massachusetts Fognet’s Field Guide to OpenView NNM Fognet’s Field Guide to OpenView Network Node Manager Second Edition Mike Peckar Copyright © 2006, 2008, Fogbooks and Mike Peckar, All Rights Reserved . Printed in the United States of Am erica, Spain, and the United Kingd om ISBN 0-9785627-2-4 N o p art of this p u blication may be r ep rod u ced or transmitted by an y m eans, electronic or m echanical, inclu d ing p hotocop ying, record ing, or any information storage an d retrieval system , w ithou t p erm ission in w riting from the au thor. All bran d names and p rod u ct names m entioned in this book are trad em arks or registered trad em arks of their resp ective comp anies. Lim it of Liability/ Disclaimer of Warranty: The p u blisher an d the au thor make no rep resentations or w arranties w ith resp ect to the accu racy or comp leteness of the contents of this w ork an d sp ecifically d isclaim all w arranties, inclu d in g w ithou t lim itation w arranties of fitness for a p articu lar p u rp ose. N o w arranty may be created or extend ed by sales or p romotional m aterials. The ad vice and strategies contained herein may not be su itable for every situ ation. If p rofessional assistance is requ ired , the services of a comp etent p rofessional p erson shou ld be sou ght. N either the p u blisher nor the au thor shall be liable for d amages arising herefrom . The fact that an organization or w ebsite is referred to in this w ork as a citation and / or p otential sou rce of fu rther in formation d oes not mean that the au thor or p u blisher en d orses the in form ation the organization or w ebsite may p rovid e or recommen d ations it may make. Fu rther, read ers shou ld be aw are that internet w ebsites listed in this w ork m ay have changed or d isa p p eared betw een the tim e w hen this w ork w as w ritten and w hen it is read . LR21 Pu blisher contact, feed back, and ord ers: ii w w w .fogbooks.com Contents at a Glance 1- Introd u ction ..........................................1 2 - General Information ............................3 3 - Com m on Issu es ..................................16 4 - N N M and Nam e Services .................25 5 - Discovery .............................................39 6 - Status Polling - netmon ......................55 7 - Status Polling - APA ..........................71 8 - Traps, Events and Alarm s ................98 9 - Interpreting Events ..........................112 10 - Event Correlation .............................123 11 - SN MP Fu nctions ..............................139 12 - System s Management ......................148 13 - Data Collection and Threshold s ....153 14 - N otifications ......................................164 15 - Fau lt Analysis Tools ........................175 16 - Third Party Tools .............................182 17 - Cisco Devices ....................................185 18 - ovw Map Operations ........................198 19 - Web Interface ....................................218 20 - Extend ed Topology ..........................225 21 - Dynam ic View s ................................240 22 - Databases and Reporting ................262 23 - Distribu ted N NM .............................275 24 - H igh Availability and Backu p .......281 25 - Firew alls ............................................291 26 - Prod u ct Details .................................295 Glossary .......................................................314 Ind ex .............................................................320 iii Table of Contents Contents at a Glance ................................ iii Table of Contents ...................................... iv 1. Introduction ............................................ 1 Acknow led gem ents .............................................................................. 2 About Fognet and the author .............................................................. 2 Conventions used in this guid e .......................................................... 2 2. General Information ............................. 3 Paths ....................................................................................................... 3 Default passw ord s ................................................................................ 3 OpenView environm ent variables ..................................................... 4 Extend ing OV environment variables ............................................... 5 N N M environm ent variables usage restrictions .............................. 5 Custom ize d aem ons via local registration files (LRF) ..................... 5 Application registration files (ARF) ................................................... 6 Sym bols, OID types and field s (FRF) ................................................ 7 Application d efaults upd ate proced ure (app -d efaults) .................. 9 Troubleshooting Wind ow s paths ..................................................... 10 Force or restart an ET d iscovery ....................................................... 10 Re-d o d iscovery (start from scratch) ................................................ 12 Logging N N M errors to Wind ow s event log .................................. 12 Running ovstart and ovpause as non -root (UN IX) ........................ 12 Com m on log files ................................................................................ 13 N N M prod uct d ocum entation .......................................................... 13 Man Pages (Linux) .............................................................................. 14 N N M patch inform ation .................................................................... 14 iv Table of Contents 3. Common Issues .................................... 15 Troubleshooting resources ................................................................. 15 Licensing issues d ue to upgrad es, m igrations, hom ing ................ 16 Changing N N M server hostnam e or IP ad d ress ............................ 16 Installation issue: virtual d irectories not created ............................ 16 Installation issue: ovas not running ................................................... 17 Installation issue (UN IX): Using N FS m ounted CDs ..................... 17 Installation issue (Wind ow s): N on-stand ard directory ................. 18 Installation issue (Wind ow s): Term inal Services ............................ 18 Unable to d ow nload JRE through firew all ...................................... 18 Devices discovered in unm anaged state .......................................... 18 Cleaning orphaned objects from object d atabase ........................... 19 Daem ons ovet_d a* not running (ovstatus) ...................................... 19 N etscape and N N M co-existence (UN IX) ........................................ 20 Cannot allocate 128 colors - using m onochrom e im ages ............... 20 Unable to load any useable font set (UN IX) .................................... 20 xterm : unable to locate a suitable font (UN IX) ............................... 21 Missing charsets in string to fontset (Solaris).................................. 21 Missing charsets in string to fontset (H P-UX)................................. 21 N TLM - passw ord need ed - firew all: unknow n ............................. 21 Troubleshooting d aem ons using ovstart and ovstatus .................. 22 Autom atically d etect and restart OV d aem ons that d ie ................ 22 Filtering on custom ovwdb field s ....................................................... 22 Working w ith m ultiple JAVA versions ............................................ 22 4. NNM and Name Services ................... 24 H ow nod e objects are nam ed ............................................................ 24 H ow preferred IP ad d ress for SN MP is chosen .............................. 26 H ow nod e objects are labeled ............................................................ 26 H ow interface objects are nam ed by netmon ................................... 27 ifDescr nam ing restrictions ................................................................ 27 v Fognet’s Field Guide to OpenView NNM H ow netw ork objects are nam ed ...................................................... 27 N od e and interface nam ing restrictions .......................................... 28 ipN oLookup.conf, snm pnolookupconf, exclud eip.conf ............... 28 General DN S consid erations ............................................................. 28 Cache-only DN S.................................................................................. 29 Split horizon DN S............................................................................... 30 BIN D-based DN S im plem entation troubleshooting ...................... 30 BIN D-based DN S on Wind ow s ........................................................ 31 “N am e services are perform ing poorly” alarm s ............................ 31 Tuning lookup perform ance (Wind ow s) ........................................ 32 Tuning lookup perform ance (UN IX) ............................................... 33 N N M d oesn‟t d etect DNS changes (Wind ow s) .............................. 34 To A record or not to A record ......................................................... 34 A record com pare issue on Solaris platform ................................... 35 nscd issues on Solaris .......................................................................... 35 N am e resolution support com m and s: ............................................. 35 N N M DN S tracing .............................................................................. 36 N am e services-related docum entation ............................................ 36 N N M in d em o, DH CP or m obile environm ents ............................ 36 5. Discovery via netmon .......................... 37 Preparing for N N M d iscovery .......................................................... 38 After N N M discovery ........................................................................ 38 Discovery process in a nutshell ........................................................ 38 Discovery polls vs. other types of netmon polls .............................. 39 Und erstanding netmon-based level 2 d iscovery ............................. 39 Lim iting d iscovery .............................................................................. 39 Externally m anaging or unm anaging objects ................................. 41 Stop d iscovery of non -IP interfaces for a d evice class ................... 42 Stop all SN MP discovery/ pollin g to a d evice class ....................... 42 Discovery hints ................................................................................... 42 Seed ing d iscovery ............................................................................... 42 vi Table of Contents Configuring m ultiple SNMP com m unity strings ........................... 43 Preferred SN MP m anagem ent ad d ress ............................................ 43 Forcing the SN MP ad d ress for a nod e ............................................. 44 netmon‟s auto-adjusting d iscovery polling algorithm .................... 45 PERL script: find OIDs not listed in oid _to_type ........................... 45 load hosts ............................................................................................... 45 Discovery of 31 and 32 bit subnetw orks .......................................... 47 Discovery of H SRP, clusters, and m ultilinked routers .................. 47 Sum m ary of netmon configuration files ........................................... 48 Und ocum ented netmon LRF sw itch settings ................................... 49 SN MP queries issued by netmon ....................................................... 49 6. Status Polling - netmon ....................... 52 ICMP and SN MP id iosyncrasies and “brow n -outs” ...................... 52 Ad justing ICMP and SNMP status polling intervals ..................... 52 netm on.noDiscover and netm on.MACnoDiscover files................ 53 netm on.interfaceN oDiscover file ...................................................... 53 Com m only used netmon LRF sw itch settings .................................. 55 netmon global status polling d efault ................................................. 55 netmon object-based polling (V6.2+) ................................................. 55 Dynam ically-adjusting status polling by netmon ............................ 55 netmon layer 2 polling algorithm s ..................................................... 56 N on-IP unconnected port status (netmon) ....................................... 57 netmon SN MP-based status for nod es .............................................. 57 Determ ine netmon-based polling intervals for a d evice ................. 58 netmon critical path analysis .............................................................. 58 N od e status events – interpretation .................................................. 59 H and ling m ultiple OV_N od e_Up events ........................................ 59 Status event variable bind ings of interest ........................................ 62 ICMP burst polls ................................................................................. 63 ICMP red irects ..................................................................................... 63 netmon vs. APA poller ........................................................................ 64 vii Fognet’s Field Guide to OpenView NNM netmon polling statistics ..................................................................... 65 netmon troubleshooting, tracing and d um ping .............................. 67 7. Status Polling - APA............................ 68 APA Architecture ............................................................................... 68 APA vs netmon status architecture ................................................... 69 APA and Sm art Plug-Ins ................................................................... 71 Turn APA polling on or off (“the big sw itch”) ............................... 71 Determ ine poller control.................................................................... 71 APA configuration in a nutshell ....................................................... 71 Dem and polling using ovet_d em and poll.ovpl .............................. 72 APA Island Group Monitoring ......................................................... 73 APA status events ............................................................................... 74 APA status from SN MP traps ........................................................... 74 Suppress/ allow APA status polling ................................................ 75 APA status events ............................................................................... 75 APA status as reflected in ovw m aps .............................................. 76 APA aggregated port support .......................................................... 77 APA H SRP and VRRP status support ............................................. 78 APA board status support ................................................................. 79 paConfig.xm l: the APA configuration file ...................................... 80 paConfig.xm l evaluation ord er issues ............................................. 81 Interface ICMP polling ....................................................................... 81 Synchronizing poll settings w ith configuration polls ................... 81 paConfig.xm l polling granularity class specifications .................. 81 Using ET topology filters to specify polling granularity .............. 82 Filter assertion type attributes .......................................................... 83 Filter nod es based on SNMP sysObjectID ....................................... 85 Force a d evice to be polled via ICMP or SN MP only .................... 86 Filtering by ifType (APA) .................................................................. 87 Disable ICMP to a firew all (APA) .................................................... 87 Sw itching routers and routing sw itches in the APA ..................... 88 viii Table of Contents Interesting sw itch interface filter exam ple ...................................... 88 APA topology events .......................................................................... 89 APA and im portant node filters ....................................................... 89 APA perform ance im provem ents ..................................................... 90 Connected vs unconnected interface APA status ........................... 90 APA m em ory footprint red uction .................................................... 91 Disabling status brid ge ....................................................................... 91 Characterizing APA polling behavior .............................................. 91 Im proving SN MP-based status w ith the APA ................................ 91 APA and Managem entAd d esss picking .......................................... 92 Troubleshooting APA ......................................................................... 93 8. Traps, Events and Alarms .................. 94 What is a trap vs. an event vs. an alarm ? ......................................... 94 Configuring SN MP traps via trap m acros ....................................... 94 Configuring events or traps via trap d efinition files ...................... 95 Configuring events or traps m anually ............................................. 95 Drop SN MP traps from particular d evices ...................................... 96 Autom atically suppress SN MP trap storm s .................................... 97 Generating ad -hoc SN MP traps or N N M events ............................ 98 Differences betw een snmpnotify and ovevent ............................... 98 Event variable bind ings ...................................................................... 99 Event logging ..................................................................................... 100 Dum ping the entire eventdb ............................................................. 101 Interpreting ovd um pevents output ................................................ 101 Dum ping parts of eventdb................................................................. 103 Ad hoc queries of the eventdb .......................................................... 103 Trap and event forw ard ing .............................................................. 103 OpenView enterprise and N N M-generated events ..................... 104 Alarm and icon status color d efaults .............................................. 105 ovalarmsrv LRF settings .................................................................... 106 Alarm brow ser settings (XN m event app -d efaults) ...................... 106 ix Fognet’s Field Guide to OpenView NNM Ad d itional actions ............................................................................ 106 9. Interpreting Events ............................ 107 N N M‟s hand ling of generic and specific traps ............................. 107 Generic traps and origin event OIDs ............................................. 107 “Received event X, N o form at in trapd .conf” ............................... 108 Agent in d istress; “spinning in ifTable or ipAd d rTable” ........... 110 OV_PhysAd d r_Mism atch ............................................................... 110 Inconsistent subnet mask................................................................. 111 OV_Duplicate_IP_Ad d ress ............................................................. 113 OV_DuplicateIfAlias ........................................................................ 113 Authentication failures .................................................................... 114 Unknow n or unrecognized ASN .1 type # m essages ................... 115 Elim inating events from und esirable system s .............................. 116 10. Event Correlation .............................. 117 What is event correlation? ............................................................... 117 Correlation Manager ........................................................................ 118 Correlation Com poser ...................................................................... 118 ECS Designer ..................................................................................... 120 Instances and stream s ...................................................................... 120 Special correlations ........................................................................... 121 Connector d ow n event correlation circuit .................................... 121 N od eIf event correlation .................................................................. 123 Repeated event correlation circuit .................................................. 124 PairWise event correlation circuit .................................................. 124 DeDup event correlation ................................................................. 125 Interm ittent Status event correlation ............................................. 125 OV poller plus event correlation .................................................... 126 Multiple reboots event correlation ................................................. 126 Cisco chassis event correlation ....................................................... 127 x Table of Contents Sum m ary of event correlations affecting status........................... 127 Useful ECS m anager com m and s ..................................................... 129 Configuring ECS circuits .................................................................. 129 Disable or enable ECS ....................................................................... 129 Logging incoming ECS events ........................................................ 130 Logging output and correlated events ........................................... 130 Sim ulate events for testing ECS logic ............................................. 130 ECS tracing ......................................................................................... 131 Stack tracing via pm d m gr ................................................................ 131 Monitoring pmd bottlenecks (N N M 6.1 only) ............................... 131 Correlation Com poser tracing ......................................................... 131 11. SNMP Functions ............................... 133 SN MP versions and history ............................................................. 133 Configuring SN MP comm unity strings on N N M agent ............. 134 Configuring SN MP versions ........................................................... 134 Configuring m ultiple SNMP agents on a single nod e ................. 134 SN MP m aster agent sw itches .......................................................... 135 SN MPv3 configuration using SN MP security pack ..................... 135 SN MPv3 Security Pack installation ................................................ 136 Configuring and controlling trap d estinations ............................. 136 SN MPv3 and VLAN inform ation ................................................... 137 Controlling the N N M server SN MP ad d ress ................................ 137 Controlling an agent‟s SN MP ad d ress ........................................... 137 Com m unity string d iscovery ........................................................... 137 CERT ad visory CA-2002-03, SN MP vulnerabilities ..................... 138 SN MP PDU size lim itations ............................................................. 138 ovtrapd d aem on not starting (Wind ow s) ........................................ 139 N N M w ith SMS or other SN MP tools (Wind ow s) ....................... 139 Bind ing SN MP trap reception to ports other than 162 ................ 139 SN MP m anager com m and line utilities ......................................... 139 Strip OID and type data from SN MP query output ..................... 140 xi Fognet’s Field Guide to OpenView NNM Perl script: output a node‟s IP ad d ress table via SN MP.............. 140 SN MP w eb resources ....................................................................... 141 12. Systems Management ...................... 142 Syslog integration facility ................................................................ 142 Managing system s w ith H OST RESOURCES MIB ...................... 144 Monitoring processes using H OST RESOURCES MIB ............... 144 Monitoring d isks using H OST RESOURCES MIB ....................... 144 Managing system s using RDMI (Wind ow s) ................................. 144 Forw ard ing Wind ow s events to N N M as SN MP traps............... 145 Forw ard ing N N M events to Wind ow s event log ........................ 145 Managing system s using Solstice enterprise agents (SEA )......... 145 Accessing WMI d ata via SN MP ...................................................... 146 N N M server system m anagem ent via SN MP (UN IX) ................ 146 Extensible SN MP agents .................................................................. 146 13. Data Collection and Thresholds .... 147 Data collection best practices .......................................................... 148 If%util (or other % expression) is greater that 100% .................... 148 MIB expressions using mibExpr.conf and mib.coerce ................ 149 MIB expression guru (ovexprguru) ............................................... 150 Und erstanding and m anipulating statistical threshold s............. 151 snmpCollect com m and s ..................................................................... 151 pingResponseTim e, p ingPercentRetry threshold events ............ 151 Data collection and d evice naming/ IP ad d ress issues ................ 152 OpenView specifics reserved for custom events .......................... 152 Data collection and high speed links ............................................. 152 Exhausted 32-bit counters w hen 64-bit is supported .................. 153 Printing graphs (UN IX) ................................................................... 153 Insufficient mem ory w hen launching graphs (Solaris) ............... 154 Exporting graphs to other im age form ats (UN IX) ....................... 154 xii Table of Contents Data collections on VLAN s .............................................................. 154 xnm graph com mand exam ples ....................................................... 154 SN MP d ata collection term inology ................................................. 155 System Insight (Com paq Insight) Manager MIBs ........................ 155 SN MP d ata collection vs. RMON .................................................... 156 Calculate epoch tim e for snm pColDum p, etc. .............................. 156 14. Notifications ..................................... 157 Configuring an action callback........................................................ 157 Using trusted Cm d s.conf file ............................................................ 157 General action callback usage consid erations ............................... 158 Metacharacters in action callbacks.................................................. 159 Selected special variables in action callbacks ................................ 159 N otification interactions w ith event correlation ........................... 159 ovactiond LRF settings ....................................................................... 160 Lim iting Action callbacks to a subset of objects ........................... 160 External File to specify action callback sources ............................ 161 Using Perl for actions callbacks ....................................................... 161 Action callbacks on Wind ow s platform s ....................................... 161 Email notifications using native tools (Wind ow s) ....................... 162 Email notifications using native tools (UN IX) .............................. 163 Email notifications and DN S issues (UN IX) .................................. 163 Aud io notifications (UN IX) ............................................................. 164 Sound notifications (Wind ow s)....................................................... 164 Exam ple notification setups............................................................. 164 Popu lar notification and paging tools ............................................ 165 Configuring an attached m od em on UN IX servers ..................... 166 15. Fault Analysis Tools ......................... 167 Problem diagnostics (PD)................................................................. 167 Sm art path (access path view ) ......................................................... 168 Find rouge MAC ad d resses ............................................................. 169 xiii Fognet’s Field Guide to OpenView NNM The ping tools .................................................................................... 169 Isolating local netw ork issues ......................................................... 170 rnetstat ................................................................................................ 170 Display port ad d ress m appings ..................................................... 170 netcheck.............................................................................................. 171 m ibtable .............................................................................................. 171 tracert and traceroute ....................................................................... 171 Packet sniffers ................................................................................... 172 Using nettl.......................................................................................... 172 16. Third Party Tools .............................. 174 Environm ental m onitoring and contact management ................ 174 Freew are utilities for UNIX ............................................................. 174 Emulating X w ind ow s on Wind ow s m achines ............................ 175 Event correlation tools ..................................................................... 175 SN MP and netw ork m anagem ent tools ........................................ 175 Topology and netw ork m od eling tools ......................................... 175 Reporting and graphing tools ......................................................... 175 17. Cisco Devices ..................................... 177 Cisco MIBs ......................................................................................... 177 CiscoWorks and N N M ..................................................................... 178 CiscoWorks, N N M, JPI and Internet Explorer ............................. 178 N N M and handling Cisco VLAN s ................................................. 179 N N M‟s Cisco discovery configuration feature ............................. 180 Setting ET to prefer CDP over FDB connection data ................... 181 Useful Cisco IOS SN MP com m and s .............................................. 181 Ad vanced Board status via the APA ............................................. 182 Com m and to create cut SN MP view s ............................................ 182 H ex target IP for Cisco ping MIB ................................................... 182 Und ocum ented IOS comm and s ..................................................... 183 xiv Table of Contents Weird trap OIDs (.1.3.6.1.6.3.1.1.5.4.1.3.6.1.4.1.9) ......................... 183 Unknow n trap 1.3.6.1.2.1.0.1&2 from Catalyst 2950 .................... 183 Cisco linkDow n trap configuration ................................................ 184 Controlling Cisco link d ow n trap variable bind ings ................... 184 Interface ind ex rem apping ............................................................... 185 ifSpeed and polling issues w ith d ata collections .......................... 185 Interesting Cisco MIBS, OIDs and MibExprs ................................ 185 What is exponentially-d ecayed m oving average? ........................ 187 Cisco tem perature probes ................................................................ 187 Directly integrating Cisco syslog m essages................................... 187 Cisco-specific event correlation circuits in N N M ......................... 188 Cisco links .......................................................................................... 188 CiscoWorks links ............................................................................... 189 18. ovw Map Operations........................ 190 Map operations – selecting, cutting and pasting objects ............. 190 Tips for using the find operation .................................................... 191 Setting unset capability field s .......................................................... 191 Map object visual cues ...................................................................... 192 Sum m ary of ovw subm ap properties .............................................. 192 N ew objects d o not show up in m aps ............................................ 193 Tw o d evices com bine into a single nod e ....................................... 193 Controlling trunked or m eshed connections in ovw .................... 194 Und erstanding sym bol and object d elete operations .................. 194 Fatal IPMAP or ovw init failed errors ............................................. 195 Rem ove a stubborn object from all maps and d atabases............. 195 Rem ove “REMOVED” objects from d atabases ............................. 195 Manually ad d objects ........................................................................ 196 Externally m anage or unm anage objects ....................................... 197 Overrid ing IPMAP sym bol changes ............................................... 197 Map status propagation rules .......................................................... 198 Sym bol status ..................................................................................... 198 xv Fognet’s Field Guide to OpenView NNM Transient, persistent and on -d em and submaps ........................... 200 setStatus utility .................................................................................. 200 Lining up sym bols in a perfect line or row ................................... 201 Forcing icons to scale d ow n in size ................................................ 201 Forcing icons to scale up in size...................................................... 202 Find ing and closing open ovw m aps .............................................. 202 IPMAP tracing ................................................................................... 202 N etw ork nam ed “10” or “arpanet” on internet subm ap ............ 203 N U0 interfaces turn red in m ap ...................................................... 203 Sym bols superim posed on each other in subm ap ....................... 203 Managing VLAN s ............................................................................. 204 Managing VPN end points ............................................................... 204 Managing ISDN interfaces .............................................................. 205 Sw itching routers and routing sw itches in ovw ............................ 205 Wireless .............................................................................................. 206 Tuning ovwdb for large num bers of objects ................................... 206 Lim iting ovw m enu bar access for som e users .............................. 206 Lim iting usage of read -w rite m aps ................................................ 206 Lim iting access to certain N N M applications ............................... 207 Manipulating m ap ow nership and perm issions .......................... 207 ovw .auth, ovw db.auth, ovspm d .auth, and ovserver files .......... 207 Wind ow s-specific m onitoring tools in N N M Menus .................. 208 Ad d right-click pop-up m enu item s to ovw GUI .......................... 208 Ad d m enu item to launch SSH on selection (UN IX) ................... 208 19. Web Interface .................................... 210 URLs ................................................................................................... 210 Maintaining open ovw map sessions for w eb clients ................... 211 Web GUI login passw ord ................................................................ 212 Web GUI access using jovw registration files ............................... 212 Ad d ing Web GUI m enu and toolbar item s ................................... 212 Icons OK in ovw but not OK in w eb GUI ...................................... 213 xvi Table of Contents javaGrapher contributed d ata collection tool ................................ 214 Running JAVA apps in event action callbacks ............................. 214 Secure JAVA-based telnet or ssh ..................................................... 214 Setting the preferred w eb brow ser ................................................. 214 Web GUI access control .................................................................... 215 Web GUI and ovw auth files ........................................................... 216 20. Extended Topology........................... 217 Enabling Extend ed Topology .......................................................... 217 ET autozoning .................................................................................... 218 Manual zoning tips ........................................................................... 218 Restart or force ET d iscovery........................................................... 219 ET d evice agents ................................................................................ 219 ET d iscovery and SN MP configuration ......................................... 219 ET d iscovery consid erations and limitations ................................ 220 Im proving ET d iscovery accuracy .................................................. 221 Im proving ET d iscovery perform ance ........................................... 221 Red ucing ovet_poll startup tim e ..................................................... 222 ET and ovw d atabase synchronization issues ................................ 223 ET single/ increm ental nod e d iscovery .......................................... 223 Detect interface configuration changes .......................................... 224 Detect interface table add itions ....................................................... 225 Zone d iscovery tips and perform ance tricks ................................. 225 Passw ord to access ET Configuration page ................................... 226 Connection ed itor .............................................................................. 226 End nod es not in sam e zone as connector ..................................... 227 Ad d connections for unsupported d evices ................................... 227 Layer 3 ed ge connectivity ................................................................ 228 Exclud e nod es from ET d iscovery .................................................. 228 Show ing d evice d etails for non -SN MP d evices ............................ 228 Sw itching routers and routing sw itches in ET .............................. 229 ET com m and sum m ary, support tools, log files ........................... 229 xvii Fognet’s Field Guide to OpenView NNM ET and DN S issues ........................................................................... 231 21. Dynamic Views ................................. 232 Dynam ic view s available ................................................................. 232 Dynam ic View s URLs ...................................................................... 233 Dynam ic View s configuration ........................................................ 234 Dynam ic View s concurrent view s/ users ...................................... 234 Dynam ic View s access via Webstart .............................................. 234 Dynam ic view s access control ........................................................ 235 Dynam ic view s via SSL .................................................................... 236 Container view s ................................................................................ 236 Container View custom ization best practices ............................... 237 Container View operations.............................................................. 238 Container View s access control ...................................................... 239 Container View exam ple ................................................................. 241 N od e View ......................................................................................... 241 Interface View ................................................................................... 242 N od e Status and Interface Status View s........................................ 242 Alarm View ....................................................................................... 243 N eighbor View .................................................................................. 243 VRRP View ........................................................................................ 243 OSPF View ......................................................................................... 244 Path View ........................................................................................... 244 H SRP View ........................................................................................ 245 Managing overlapping ad d ress spaces (OAD) ............................ 246 Delete an overlapping ad d ress spaces (OAD) .............................. 247 IPV6 .................................................................................................... 248 Port Ad min tool ................................................................................ 249 Aggregated port support ................................................................. 249 Increase JAVA heap size .................................................................. 250 Ad d , d elete, m anage or unm anage objects via URLs .................. 250 Troubleshooting d ynam ic view s .................................................... 251 xviii Table of Contents Dynam ic view s upd ate issues d ue to caching .............................. 251 Dynam ic view s registration files..................................................... 252 Dynam ic view s and ed ge connectivity .......................................... 252 Transfer MIB application build er apps to Dynam ic view s ......... 252 Custom ize d ynam ic view s-based alarm brow ser ......................... 253 22. Databases and Reporting ................ 254 Data w arehouse ................................................................................. 254 Solid d atabase .................................................................................... 255 Data w arehouse queries ................................................................... 255 Solid d atabase perform ance............................................................. 257 Determ ining the version of Solid .................................................... 257 Database com m and utilities ............................................................ 257 Reporter URLs ................................................................................... 258 Connecting Crystal Reports to N N M (Wind ow s) ........................ 258 Using Oracle for the d ata w arehouse ............................................. 258 ODBC Driver Manager Version errors........................................... 259 Accessing DW d ata d irectly from Microsoft Excel ....................... 260 Data w arehouse com patibility: N N M 6.0 to N N M 6.1 ................ 260 Disable the data w arehouse ............................................................. 260 Rebuild the d ata w arehouse ............................................................ 260 Rebuild the ET Database .................................................................. 261 Disabling N N M 6.2 d efault d ata collections ................................. 261 Extend and custom ize router perform ance reports ..................... 262 Extend ing the num ber of instances in the TopN report .............. 262 Support and contributed reporting tools ....................................... 262 Database m aintenance ...................................................................... 262 Delete old reports .............................................................................. 263 Rebuild DW after trim to recover space (UN IX) .......................... 263 Resolving errors w ith d atabase m aintenance program s ............. 264 Extrem e filtering in event d ata exports .......................................... 264 Database size lim itations .................................................................. 265 xix Fognet’s Field Guide to OpenView NNM Increase size of the binary event store (BES) ................................ 265 Clear the event d atabase (BES) ....................................................... 265 Dum p list of managed nod es into CSV file (UN IX only) ............ 265 Dum p a clean list of all m anaged nod es (UN IX only)................. 266 Dum p interface list for all nod es (UN IX only) ............................. 266 23. Distributed NNM ............................. 267 DIM limitations ................................................................................. 267 DIM overlap m od es .......................................................................... 268 Ports used by MS and CS:................................................................ 268 Polling through firew alls using DIM ............................................. 268 Filtering .............................................................................................. 268 ovfiltercheck, ovfiltertest and ovtopod um p ................................. 269 Exam ples of filter expressions ........................................................ 269 Using external files w ithin filters ................................................... 270 Filtering id iosyncracies .................................................................... 270 Exam ples of xnm topoconf com m and s .......................................... 271 Rem ove second ary objects from the d atabase .............................. 271 DIM set-up in a nutshell .................................................................. 271 Migrating to non-DIM w ith APA ................................................... 271 Using WEB interface w ith an MS as Managem ent Console ....... 272 CS view s from MS Alarm Brow ser ................................................ 272 24. High Availability and Backup ....... 273 Autom ated d atabase backups ......................................................... 273 ovbackup and OpenView d ata protector (Om niBack) ................ 274 ovresum e tim es out .......................................................................... 274 Running ovbackup.ovpl as non -root user (UN IX)....................... 274 ovbackup.ovpl and Data Warehouse interaction ......................... 274 ovdbcheck daem on not starting (MS SQL) ...................................... 275 Backing up SN MP configuration d ata ........................................... 275 xx Table of Contents Integrating N N M w ith OVO d atabase backup ............................. 275 Backing up the Solid d atabase ......................................................... 275 Map exports and im ports ................................................................. 276 Map snapshots ................................................................................... 276 ovtopod bsnapshot.ovpl .................................................................... 277 N ative backup and restore com m and s (UN IX) ............................ 277 DIM Configuration for a hot stand by system ............................... 277 N on-DIM configuration for a “hot stand by” ................................ 278 N N M as a highly available service ................................................. 280 Syslog in an H A environm ent ......................................................... 281 N N M and m ultiple N IC card s ........................................................ 281 Managing migratable IP ad d resses ................................................ 281 25. Firewalls and Security ..................... 283 N N M Ports used ................................................................................ 283 Reconfiguring ports used by N N M ................................................ 284 H ighly-secure netw ork m anagem ent scenario ............................. 284 N N M ICMP polls versus ping sw eep attacks ............................... 284 Security issue w ith ICMP pin g (UN IX only) ................................. 285 Discovering into subnets using NAT ............................................. 285 Monitoring DMZ d evices ................................................................. 285 Extend ed Topology configuration GUI passw ord ....................... 286 26. Product Details .................................. 287 N N M 8i ............................................................................................... 287 N N M 8i architecture ......................................................................... 288 Prod uct feature d eltas by N N M version ........................................ 290 Prod uct feature d eltas by OS ........................................................... 295 OS and softw are support m atrix ..................................................... 296 Stand ard ed ition vs. Ad vanced edition ......................................... 298 N od e d efinition ................................................................................. 298 N N M sm art plug-ins ........................................................................ 300 xxi Fognet’s Field Guide to OpenView NNM H P Softw are prod uct suite sum m ary ............................................ 300 Insight m anager integration ............................................................ 301 Core N N M prod uct num bers (U.S), versions 6.4-7.51................. 302 Support package num bers (N orth Am erica only) ....................... 304 Glossary.................................................... 305 Index ......................................................... 311 xxii 1. Introduction This vade mecum is w ritten for field consu ltants, u sers and ad m inistrators of the H P N etw ork N od e Man ager (N N M) softw are p rod u ct. The second ed ition covers inform ation that is relevant for N N M p rod u ct versions 6.0 throu gh 7.53 w ith Interm ed iate Patch 20. It d oes not cover N N M 8i, w hich is an entirely new and d ifferent p rod u ct (see p age 287). It w as w ritten for those w ho seek a shortcu t to com m only u sed p rod u ct info that is either m issing or obfu scated in the p rod u ct d ocs, and it covers p ractical im p lem entation inform ation that can‟t be fou nd in any p rod u ct d ocu m entation or the fine p rod u ct m anu al or reference p ages. This gu id e w as gleaned from Op enView u sers and from the au thor‟s fifteen years of com p iled notes on the p rod u ct. So, w hat is N N M? N N M is a scalable enterp rise netw ork fau lt m anagem ent tool that p rovid es SN MP m anager featu res, netw ork d iscovery, OSI layer 2 and layer 3 top ology m ap p ing, netw ork d evice statu s p olling, third p arty integration APIs and som e lim ited rep orting, p erform ance and trend analysis cap abilities. It is the m ost p op u lar enterp rise-class netw ork m anagem ent tool for m anaging IP netw orks. The term s “Op enView ” and “N N M” have been u sed synonym ou sly for m any years, bu t it has never been accu rate to m ix these term s. N N M is a d iscreet p rod u ct w ithin the Op enView softw are p rod u ct su ite. A list of som e of the m ore p op u lar and relevant H P Softw are p rod u cts are listed on p age 300. Shortly after its acqu isition of Mercu ry Interactive in 2007, H P d rop p ed the u se of “Op enView ” in all its softw are p rod u cts and brand ed all softw are u nd er the H P u m brella “H P Softw are.” “H P Op enView N etw ork N od e Manager” thu s becam e “H P N etw ork N od e Manager.” The au thor‟s exp ectation is that N N M w ill still be referred to as “Op enView ” for at least another d ecad e. The second ed ition contains exp ansions, corrections and coverage of new featu res released in N N M 7.51 and 7.52 and 7.53 and interm ed iate 1 Fognet’s Field Guide to OpenView NNM p atches. Alm ost all acronym s u sed w ithin this gu id e are exp and ed in an attem p t at a com p rehensive glossary starting on p age 305. Feed back on this gu id e is w elcom ed and encou raged and can be su bm itted to the au thor throu gh the p u blisher. Acknow ledgements Rachel p roved ind isp ensable as chief ed itor for this book. The au thor‟s sister Su e Wied orn d id the w ond erfu l cover art. The au thor‟s child ren, Molly, Jake and Charlie su ffered greatly from a severe lack of attention d u ring the w riting and u p d ating of this book. Sp ecial thanks for inp u t nd into the 2 ed ition go to Kevin May. Tu sen takk to N ils Johannessen for p rovid ing cover to cover ed iting for the 2nd ed ition and contribu ted som e great ad d itional m aterial. Kevin Sm ith of H P has p rovid ed valu able insights for both ed itions. Finally, Tracy Avent of H P p rovid ed both technical and m oral su p p ort and is ju st to o nice of a fellow to skip a cred it herew ith. Mu ch of this w ork w ou ld have been im p ossible w ithou t the m entorship of som e of the eld er statesm en and statesw om en of the Op enView Foru m listserv and greater Op enView com m u nity. In 2007, OVForu m w as renam ed / rebrand ed to VivIt (vivit-w orld w id e.org) About Fognet and the author Mike joined Digital Equ ip m ent Corp in 1987 and in 1993 began w ork on the DEC/ IBM OEM‟d p orts of N N M 3.31 to Alp ha OSF/ 1 and N T/ Alp ha called PolyCenter and Tivoli N etView . Follow ing that, Mike w orked for H P as a senior p ost-sales Op enView consu ltant. H e fou nd ed Fognet in 1998 as a consu ltancy focu sed on enterp rise m anagem ent. Fognet is an IT consu lting service com p any, d elivering enterp rise m anagem ent p rod u ct im p lem entations, training, architectu re and integration w ith a p rim ary focu s on the Op enView N N M and Op erations p rod u cts. Visit Fognet‟s w eb site at w w w .fognet.com . Conventions used in this guide Backgrou nd and foregrou nd p rocesses are listed italics. Com m and s are listed in Courier font. File nam es are listed in regu lar font. 2 2. General Information This section covers som e basic inform ation and p roced u res w hich are m ost frequ ently u sed to ad m inister N N M. Paths The p aths to files u sed in this gu id e are show n u sing N N M stand ard environm ent variables. The convention in this gu id e is to m ix UN IX and Wind ow s-like p aths w ith the u nd erstand ing that the read er w ill m ake the ap p rop riate translation for that read er‟s p latform . Thu s the UN IX p ath for $OV_CON F/ C/ trap d .conf, w hen u sed in this gu id e is the equ ivalent of %OV_CON F%\ C\ trap d .conf on the Wind ow s p latform . For Wind ow s p latform u sers, the u se of environm ental variables help s interp ret p aths w ithou t care as to the d irectory in w hich N N M w as originally installed . Und er Wind ow s, all N N M d irectories are relative to the installation d irectory. This is n ot the case w ith the UN IX p aths, w here p er OSF conventions; files are laid ou t as follow s: / etc/ opt/ OV/ share / var/ opt/ OV/ share / opt/ OV Configuration files Files that “grow” Optional or program files D efault passw ords The follow ing p assw ord s are u sed to access variou s N N M su bsystem s: SN MP read com m unity string: Web GUI user: Web GUI passw ord : Data Warehouse DB user: Data Warehouse DB password : public ovuser ovuser ovd b ovd b The Extend ed Top ology configu ration GUI p assw ord is in: $OV_AS\ w ebapps\ topology\ WEB-IN F\ d ynam icView sUsers.xm l 3 Fognet’s Field Guide to OpenView NNM OpenView environment variables To activate the environm ent variables on Wind ow s so they can be u sed to find d irectories from the com m and line, enter: ov.envvars.bat The absolu te p ath shou ld n‟t be necessary since the Op enView bin d irectory is p laced in the p ath u p on p rod u ct install. On UN IX, u se the ap p rop riate shell-sp ecific scrip t in the N N M bin d irectory. The follow ing scrip ts can be called from the ap p rop riate com m and line shell or from other scrip ts that them selves invoke these shells: ov.envvars.sh ov.envvars.csh ov.envvars.pl The com p lete list of environm ental variables can be fou nd w ithin the files above. Selected environm ent variables for UN IX inclu d e: OV_BACKGROUN DS OV_BIN OV_BITMAPS OV_CON F OV_CON TRIB OV_DB OV_DOC OV_FIELDS OV_H ELP OV_JRE OV_LIB OV_LOG OV_LRF OV_MAN OV_N EW_CON F OV_REGISTRATION OV_SH ARE_LOG OV_SN MP_MIBS OV_SYMBOLS OV_WWW OV_AN ALYSIS_CON F OV_SH ARE_H TDOCS OV_WWW_REG OV_AS 4 / / / / / / / / / / / / / / / / / / / / / / / / etc/ opt/ OV/ share/ backgrounds opt/ OV/ bin etc/ opt/ OV/ share/ bitmaps etc/ opt/ OV/ share/ conf opt/ OV/ contrib var/ opt/ OV/ share/ databases opt/ OV/ doc etc/ opt/ OV/ share/ field s var/ opt/ OV/ share/ help opt/ OV/ jre/ jre1.4 opt/ OV/ lib var/ opt/ OV/ share/ log etc/ opt/ OV/ share/ lrf opt/ OV/ man opt/ OV/ new config etc/ opt/ OV/ share/ registration var/ opt/ OV/ share/ log var/ opt/ OV/ share/ snm p_m ibs etc/ opt/ OV/ share/ sym bols opt/ OV/ ww w etc/ opt/ OV/ share/ conf/ analysis var/ opt/ OV/ w w w/ htd ocs etc/ opt/ OV/ share/ w w w/ registration opt/ OV/ tom cat/ jakarta-tom cat-4.0.4 General Information Extending OV environment variables There are no restrictions to ad d ing cu stom variables to the shellsp ecific scrip t in $OV_BIN/ov.envvars.* on either the UN IX or Wind ow s server nod es. N ote that Wind ow s environm ental variables are not case sensitive w hereas UN IX environm ental variables are. N N M environment variables usage restrictions Environm ent variables cannot be u sed in action callbacks. Environm ent variables can be u sed in tru sted Cm d s.conf files. Customize daemons via local registration files (LRF) Local registration files are u sed to set op tions and behaviors for Op enView ‟s backgrou nd d aem ons. LRFs are read by the com m and s ovaddobj and ovdelobj to register or d eregister an N N M m anaged p rocess w ith ovspmd. LRF changes that corresp ond to d aem on com m and line op tions are fou nd betw een the second and third set of colons in the file. So in the follow ing ovw d b.lrf file line, the com m and line argu m ent is “ -O”. OVs_YES_START::-O:OVs_WELL_BEHAVED:15:PAUSE When u p d ating an LRF, the general p roced u re to m ake the change effective is: Backup the original LRF Ed it the LRF and enter changes Run: ovaddobj $OV_LRF/<daemon>.lrf ovstop <daemon> ovstart <daemon> A com m on error is that either the ovaddobj com m and is not ru n from the $OV_LRF d irectory, or the fu ll p ath to the LRF file is not sp ecified . Som e Op enView d aem ons have d ep end encies on each other, so w hen an ovstop is issu ed for a p articu lar d aem on, several other d aem ons m ay also be shu t d ow n. Dep end ent d aem ons are listed in the second colon -sep arated field in the LRF file. In general, it is a good p ractice to stop all u ser GUI sessions w hen issu ing ovstop com m and s. To m ake su re that all 5 Fognet’s Field Guide to OpenView NNM ap p rop riate d aem ons are ru nning after starting or starting any d aem ons, ru n ovstatus –c. Debu g any p rocess configu red in a LRF by u sing the -debug x op tion in the forth LRF field . “x” as a valu e betw een 1 and 5 w ith increasing levels of verbosity. A history of ovaddobj and ovdelobj op erations is held in the $OV_CON F/ ovsu f file. In som e cases, this file can grow large, as in w hen ovaddobj and ovdelobj op erations are u sed in u ser-cu stom ized sched u led op erations to control N N M behaviors like d iscovery, etc. In this case, the ovsu f shou ld be cleared ou t on occasion to im p rove p erform ance. When d oing this, p reserve the one line low est in the file for each d aem on listed . Application registration files (ARF) Any file p laced in $OV_REGISTRATION / C is read by the ovw p rocess to register ovw GUI m enu -bar item s. The syntax is d escribed in d etail in the OVw RegIntro m an/ ref p ages. Use these sim p le ru les to gu arantee su re sm ooth sailing: 1. 2. 3. 4. Make backup copies of registration files to be m od ified , but do not store the backups in the registration d irectory tree as they w ill be read and cause errors Any filenam e supported by the OS is read by ovw Test syntax w hen m aking changes using: regverify –arf <arf filename> See page 206 for proced ure to create alternate ARF trees The Menu Bar item attribu tes in ARF files inclu d e: Selection cap abilities based on object DB field s, H otkeys, Preced ence nu m ber, and Selection ru le sp ecifications. The MIB Ap p lication Bu ild er is frontend ARF file generator d esigned to w ork w ith SN MP MIB valu es. The follow ing d irectory is reserved for MIB ap p lication bu ild er ap p s: $OV_REGISTRATION / C/ ovm ib For som e excellent exam p les, see the N N M m anu al: Creating_and_Using_Reg_Files.pd f 6 General Information Symbols, OID types and fields (FRF) oid _to_sym p rovid es the m ap p ing of d evice typ es to the sym bol u sed to rep resent those d evices. oid _to_typ e and H Poid 2typ e p rovid e the m ap p ing of d evice typ es to how to treat the d evices in the IPMAP top ology. In version 7.0 and greater, the oid _to_sym _reg d atabase w as introd u ced to facilitate better integration w ith N N M ET. For backw ard com p atibility, the oid _to_sym configu ration file is still read first w hen IPMAP starts. Field registration files (FRFs) d efine all the field s in the object (ovwdb) d atabase. FRF u p d ates are necessary w hen a new sym bol m ap p ing is d esired or to ad d u n iqu ely id entifying attribu tes to the object d atabase. The oid _to_typ e and H Poid 2typ e m ap p ings u se tw o object d atabase field valu es: “vend or”, and “SN MP agent”, to assign top ology ru les for d evices w hich m ap to those field s. For alm ost all N N M d ep loym ents, there w ou ld p robably be agents and vend ors that are not listed in oid _to_typ e, so it m ay be d esirable to ad d these m issing valu es via the FRF p rocess. If there is a need sim p ly to m ake su re d evices for an u nd efined SN MP agent are m ap p ed p rop erly in IPMAP, it is not requ ired that FRFs be u p d ated . A shortcu t is to assign p re-existing Vend or and SN MP agent valu es to the new ly ad d ed SN MP agent OID in the oid _to_typ e files that is alread y associated w ith a sym bol that satisfactorily rep resents the object. The only p roblem w ith this shortcu t is that object and top ology d atabase searches and rep orts m ight p rod u ce incorrect resu lts. When ad d ing a new entry, d eterm ine if the vend or and / or agent is alread y in the object d atabase (a p re-requ isite) by p erform ing find op erations in the ovw GUI. Alternatively on UN IX, issu e follow ing tw o com m and s: ovobjprint -e `ovobjprint -f |grep vendor |cut -f1` ovobjprint -e `ovobjprint -f |grep SNMPAg |cut -f1` Su m m ary of oid _to_typ e and H Poid 2typ e top olog y attribu tes: B D S N d Brid ge Cisco H SRP (ip ad d r‟s not in ip.ipAd d rTable) 2ndary ad d rs. Do not d elete 2ndary If‟s Ignore unnum bered interfaces on these d evices Dorm ant oper status - Cisco ISDN 7 Fognet’s Field Guide to OpenView NNM G H I L M T U Gatew ay H ub Ignore SN MP Allow Loopback con nections N ot a Gateway Term inal Server Discover as unmanaged The follow ing global d efau lts for oid _to_typ e and H Poid 2typ e are u sed to sp ecify global top ology treatm ent for classes of d evices that d on‟t otherw ise m ap to sp ecific OID entries in the file: DEFAULT_SN MP -- Supports SN MP but d oes not have a more specificOID entry DEFAULT_IP -- Supports IP and otherw ise w ould not have a matching entry DEFAULT_IPX -- Supports IPX and otherw ise would not have a m atching entry DEFAULT -- Any nod e w hich otherw ise w ould not have a matching entry For exam p le, to instru ct N N M to d iscover SN MP-su p p orted nod es that are not listed in the files as u nm anaged , u se: DEFAULT_SN MP:::U To u nm anage Wind ow s servers and / or w orkstations w ith SN MP tu rned on, change these entries: 1.3.6.1.4.1.311.1.1.3.1:Windows NT:U # Pre NT SP2 1.3.6.1.4.1.311.1.1.3.1.1:Workstation:Windows NT:U 1.3.6.1.4.1.311.1.1.3.1.2:Server:Windows NT:U 1.3.6.1.4.1.311.1.1.3.1.3:PDC:Windows NT:U To ad d a “vend or” FRF: Ed it the file $OV_FIELDS/ C/ ovw _field s and ad d the new vend or, an exam p le of the section follow s : Field "vendor" { Type Enumeration; Flags capability, general, locate; Enumeration "Unset", "Hewlett-Packard", "3Com", "ACC", "Allied Telesyn" To ad d an “SN MP agent” FRF: Ed it the file $OV_FIELDS/ C/ snm p _field s and ad d the new agent, an exam p le of the section follow s: 8 General Information Field "SNMPAgent" { Type Enumeration; Flags capability, general, locate; Enumeration "Unset", "HP 3000/XL", "HP 386", "HP 700/[R]X X-Terminal", "HP 9000/HP-UX", "HP Bridge", To m ake su re no syntax errors have been introd u ced , ru n the follow ing com m and and look for errors: Wind ow s: UN IX: regverify ovw –verify >/dev/null Once syntax has been verified , u p d ate the object d atabase u sing: ovw -fields Once valid Agent and Vend or field s are registered , the new field entries can be u sed in the oid _to_typ e file and new ly-d iscovered d evices shou ld be m ap p ed accord ing to the ru les. To u p d ate alread y d iscovered objects w ithou t having to red iscover them , exit ovw and ru n: ovstop netmon ovtopofix -u -o <sysObjectID> ovstart netmon To ad d cu stom ized sym bols, see the p roced u re “Create a Sym bol Registration File” in the Ap p end ix of th e N N M Gu id e: Managing You r N etw ork. Application defaults update procedure (app-defaults) UN IX: Ed it the ap p rop riate file in the $APP_DEFS d irectory. Wind ow s: Ed it the ap p rop riate registry key u nd er: H KEY_LOCAL_MACH IN E if the value is global for all users, or : H KEY_CURREN T_USER, if d ifferent users need to be able to use d ifferent application d efault values. The assorted classes are fou nd u nd er: H KLM\ SOFTWARE\ or H KCU\ SOFTWARE\ 9 Fognet’s Field Guide to OpenView NNM \ Hew lett-Packard \ OpenView \ Netw ork N od e Manager The ap p -d efau lts files (UN IX) or classes (Wind ow s) are: OVw XN m XN mappmon XN m events XN m graph XN mtrap ovw GUI settings, e.g. Map colors, fonts GUI settings for MIB brow ser, collector, etc GUI settings for tables and form s GUI settings for alarm brow ser GUI settings for grapher GUI settings for event configuration Troubleshooting Window s paths In N N M event configu ration action callbacks, backslashes m u st be escap ed and sp aces m u st be qu oted , for exam p le: C:\\‖Program Files‖\\‖HP OpenView‖\\program.bat In N N M filter files, the colon m u st be escap ed and the backslashes rep laced w ith regu lar slashes. servers "servers" {d\:/ov/conf/coreservers.txt} In N N M ARF files, qu otes are need ed for sp aces and m u st be escap ed , bu t all the follow ing exam p les w ill w ork: Application "Wordpad" { Command "C:/\"Program Files\"/Windows\" \"NT/Accesories/wordpad";} Application "wordpad" { Command "\"C:/Program Files/Windows NT/Accesories/wordpad\""; } Application "webpage" { Command "C:\\PROGRA~1\\ INTERN~1\\iexplore.exe http://webpage.html" } Finally, if none of the above seem s to w ork, ad d the d irectory location of the execu table to the system p ath, or u se the 'Ap p Path ' registry key, then sim p ly call the execu table nam e w ithou t sp ecifying the p ath. Force or restart an ET discovery Before attem p ting this op eration, check to see if a d iscovery is cu rrently ru nning: from H om e Base, select the Discover y Statu s tab. Check d iscovery statu s from the com m and line by ru nning: ovstatus –v ovet_disco Restart or force ET d iscovery only w hen u nd erlying configu ration changes have been m ad e that m ay requ ire su ch an action. Make su re that any d evices that m ay be the target of a force ET d iscovery are p rop erly d iscovered by netmon first. 10 General Information ET d iscovery can be initiated from H om ebase: select Discovery Statu s, then ET Configu ration, then Initiate Fu ll Discovery N ow . This can also be accessed from ovw via: Op tions->Extend ed Top ology Configu ration. Or, the w eb URL is: http://<nnm server>:7510/topology/etconfig To force ET Discovery from the com m and line, ru n: etrestart.ovpl or: etrestart.ovpl -verbose –disco Be p rep ared to w ait a w hile for d iscovery to com p lete. The p rogress of d iscovery can be follow ed on the Discovery Statu s p age. If changes have been m ad e that requ ire an ET red iscovery, and a d iscovery is alread y in p rogress, u se: etrestart.ovpl -force To refresh H om ebase and APA statu s view s, ru n: ovstop ovas ovstop ovet_poll ovstart ovas ovstart ovet_poll By d efau lt, ET w aits u ntil netmon has fou nd 2500 changes in a netw ork before ET ru n s a new d iscovery. To change this so it ru ns at a sched u led tim e, go the ET configu ration (see above) and tu rn off “Enable d iscovery for a sp ecified nu m ber of N N M changes.” Then tu rn on “Enable recu rring d iscovery.” Sched u le a d iscovery d aily, and click Ap p ly. N ote that low ering the threshold m ay have a p erform ance cost. If ET red iscovery isn‟t p rop erly d iscovering p articu lar d evices, see the section on im p roving ET d iscovery on p age 221. 11 Fognet’s Field Guide to OpenView NNM Re-do discovery (start from scratch) Re-d o d iscovery after m aking m ajor changes su ch as sw itching from files-based looku p s to DN S, or to recover from su sp ected corru p tion of the object, m ap , top ology, or ET d atabases. Before p erform ing this op eration, consid er exp orting m ap s to save cu stom izations (see p age 276). Review any ad d itional changes that m ay p rofou nd ly affect top ology, su ch as reclassifying the treatm ent of a m ajor class of connector d evices by OID, DN S changes, etc. To ju st d elete the d atabases, ru n the follow ing com m and : $OV_CONTRIB/deleteOVDB/deleteOVDB.ovpl To m anu ally start from scratch: 1. 2. 3. 4. 5. 6. 7. 8. 9. Im portant: close all OpenView GUIs ovstop –c Remove the databases & log files: UNIX: cd $OV_DB; rm –rf openview; rm –rf nnmet Wind ow s: d elete all the d irectories below : \ databases\ openview \ databases\ nnm et If exporting topology, run : ovtoposql -R i|o -C Clear the SN MP cache: xnmsnmpconf -clearCache Restart all the OV daem ons: ovstart -c Start the user interface: ovw & Re-initiate ET: setupExtTopo.ovpl Logging N N M errors to Window s event log The ovtracelog u tility is p rovid ed for logging N N M excep tions throu gh the Wind ow s tracing and logging facility. See the ovtracelog reference or m an p age for d etails. Running ovstart and ovpause as non-root (UN IX) Use the follow ing step s to allow non -root u sers to issu e the ovstart and ovstop as w ell as ovpause or ovresume com m and s. 1. 12 Make sure root is the owner of ovstart and ovstop General Information 2. 3. 4. 5. 6. 7. 8. 9. chmod 4555 ovstart ; chmod 4555 ovstop Create a file called $OV_CON F/ ovstart.allow chmod 400 ovstart.allow ; chown root ovstart.allow In the file put a user nam e or UN IX user ID chown root:sys ovstart ovstop ovstatus chown root:sys ovpause ovresume chmod 666 $OV_LOG/ovstart.log Consult the $OV_LOG/ ovstart.log for possible errors. Common log files The follow ing are com m on log files is $OV_LOG: Setup.log ovactiond .log ovalarm srv.trace ovcapsd .log ovrepld .log ovbackup.log Pm d .log0 snm pCol.trace ovdw *.log Troubleshoot installation (Window s) Troubleshoot automatic actions Troubleshoot xnm events Troubleshoot DMI/ RDMI (Window s) Troubleshoot MS-CS upd ates Troubleshoot ovbackup.ovpl actions Troubleshoot postmaster daemon Troubleshoot raw data collection Troubleshoot data warehouse and reporting The follow ing are com m on log files is $OV_PRIV_LOG: httpd *.log ovas_err.log ovrequestd.log Troubleshoot w eb server Troubleshoot Dynam ic View s Troubleshoot Data Warehouse feed s N N M product documentation User m anu als are fou nd in: $OV_WWW/ htdocs/ C/ Manuals Online help : Unix: http:/ / <nnm server>:3443/ OvCgi/ OvWebH elp.exe Wind ow s: http:/ / <nnm server>/ OvCgi/ OvWebH elp.exe Und er Wind ow s, the online help facility can be invoked d irectly even if N N M is not ru nning: 13 Fognet’s Field Guide to OpenView NNM %OV_H ELP\ C\ NN M.hlp Dow nload u ser m anu als: http:/ / ovw eb.external.hp.com/ lpe/ doc_serv Ad d itional ECS Manu als: Selected RFC‟s: White Pap ers: Release N otes: Release N ote Up d ates (V7.5): $OV_DOC/ ECS/ C $OV_DOC/ RFC $OV_DOC/ WhitePap ers $OV_WWW / htd ocs/ C/ ReleaseN otes/ http:/ / ovw eb.external.hp.com/ nnm / N N M7.5/ relNoteUpd/ relNoteUpd ate.htm Man Pages (Linux) Man p ages for d aem ons and certain com m and s are p laced u nd er / op t/ OV/ m an/ m an1m for N N M and / op t/ OV/ m an/ m an1m .Z for ECS tools. The 1m and 1m .z m anu al sections m ay not be d efined by d efau lt u nd er Linu x varients. Sp ecify the section in the com m and line to get arou nd this issu e, for exam p le: man 1m netmon man 1m.Z ecsmgr Alternatively, 1m and 1m .Z m ay be ad d ed to the MAN SECT variable in the / etc/ m an.config file, for exam p le: MAN SECT 1:8:2:3:4:5:6:7:9:tcl:n:l:p:o:1m :1m .Z N N M patch information Em ail notifications for p atch u p d ates: http:/ / support.openview .hp.com/ em ailReg.jsp Determ ining w hat p atches are installed : Wind ow s: UN IX: 14 Control Panel-Ad d / Rem ove Program s sw list -l fileset | grep "^ # PH " 3. Common Issues This section covers som e of the m ore com m on issu es encou ntered d u ring initial installation and setu p -u p of N N M. Troubleshooting resources The follow ing tool can be u sed on any N N M p latform to check p atch levels. Alw ays try to keep N N M u p -to-d ate on p atch levels: $OV_SUPPORT/checkPatch.ovpl The m ost com p rehensive d atabase of solu tions to com m on N N M p roblem s is located at the H P IT Resou rce Center at: w w w .itrc.hp.com Using the ITRC-em bed d ed search engine, how ever, is often slow and yield s p oor resu lts. Fortu nately, the entire ITRC d atabase is m ined by Google, so w hen an error occu rs that is very generic, ad d ing “ITRC” to the search term in Google can often help narrow d ow n the issu e. Another excellent resou rce is the VivIt (form erly OVForu m ) listserv at: w w w .vivit-w orld w id e.org (form erly w w w .ovforum .org ) The foru m p rovid es a few sp ecial interest grou p s, bu t by far, the m ost active is the general foru m w hich covers all Op enView p rod u cts. In p ractice, m ost of the traffic p ertains to N N M. The ovforu m listserv is m aintained by the Op enView Foru m International u sers grou p , w hich is an ind ep end ent organization that has no d irect ties to H P. For this reason, it is a good p lace to su bm it qu estions that are less biased by H P. For exam p le, qu estions p ertaining to com p araisons of Op enView w ith com p eting p rod u cts or d iscu ssions abou t the qu ality H P p rod u cts and services. Also, u ser exp eriences and cu stom izations are m ore likely to be fou nd on the listserv. Conversely, very few H P em p loyees p articip ate in the ovforu m listserv . Qu estions p osed to the ITRC, 15 Fognet’s Field Guide to OpenView NNM how ever, are very likely to be answ ered d irectly by H P su bject m atter exp erts. Licensing issues due to upgrades, migrations, homing License u p d ates are requ ired circu m stances: from H P u nd er the follow ing When upgrad ing from 6.x to 7.x When changing IP ad d ress When changing server OS If an N N M server has m ore than one IP ad d ress, the p referred IP ad d ress can be sp ecified u sing the N N M_IN TERFACE keyw ord in the follow ing file: $OV_CON F/ ov.conf Changing N N M server hostname or IP address Generally, changing the N N M server nam e or IP is som ething to avoid , m ostly becau se N N M registers itself (if it can) as a trap recip ient on SN MP agents it d iscovers, bu t also for all the reasons listed in the section on N N M and N am e Services (see p age 24). Changing IP ad d ress m eans that trap s m ay be lost, p articu larly if trap d estinations have been set m anu ally for m any d evices. Be aw are that on m any d evices, if ad d ing a trap recip ient by FQDN , this is converted to the IP ad d ress in the d evices configu ration. An IP ad d ress change also requ ires that the p rod u ct license be u p d ated . To change the hostnam e and/ or IP ad d ress: 1. 2. 3. 4. 5. 6. 7. 8. Stop NN M processes using ovstop Change the nam e in: $OV_DB/ openview / ovw d b/ ovserver Change the nam e in all the $OV_CON F/ *auth* files Change the server name in the OS Run: xnmsnmpconf -clearC Update license if IP Ad d ress changed Update agent trap d estinations if IP ad d ress changed Start processes using ovstart Installation issue: virtual directories not created Sym p tom s are that “the p age cannot be fou nd ” or “Und er constru ction” ap p ears w hen the w eb GUIs first lau nched . This hap p ens on Wind ow s installations w hen N N M 7+ is installed into an environm ent w here anti-viru s softw are p revents VBscrip ts from ru nning. 16 Common Issues To resolve this issu e, d isable any p rogram s that block vbscrip ts then locate the setupIIS.vbs scrip t from the new config su bd irectories on the installation m ed ia, then ru n: cscript setupIIS.vbs "C:\Program Files\HP OpenView\NNM" – If issu es p ersist, installation registry file entries m ay need to be u p d ated . There are articles w hich d escribe the u p d ate p roced u re on ITRC. Installation issue: ov as not running Typ ically, the root cau se is w ith the w eb server‟s configu ration. The re are m any p ossible things that can go w rong. In trou bleshooting, alw ays look to the w eb server‟s health first. Is it ru nning? Does the event log (or syslog) contain p ertinent inform ation? Do the w eb server logs look O.K.? Check the ovas.log file. ovas is the d aem on that serves d ynam ic view s. If ovas.log show s an error like this: ld.so.1: ovas: fatal: relocation error: file /opt/OV/jre/jre1.4/lib/sparc/libjava.so: symbol VerifyFixClassname: referenced symbol not found The p roblem is that LD_LIBRARY_PATH is not set correctly, p ossibly set by another ap p lication (e.g. Checkp oint). Reset it to: / opt/ OV/ lib Another com m on p roblem cau sing ovas to fail to start is that another ap p lication (p robably one that u ses Tom cat) has grabbed the Tom cat server p ort. In N N M V7.01+ this is p ort 8005. To resolve, change the Server Port nu m ber from 8005 to som e other u nu sed p ort nu m ber, then restart the ovas d aem on. The Server Port is sp ecified in the follow ing file: $OV_AS/ conf/ server.xm l Installation issue (UN IX): Using N FS mounted CD s When exp orting a rem ote CD d rive to be u sed for installing N N M, 'root' access to the d rive m u st be granted . Use a line sim ilar to the follow ing on the m achine that has the CD d rive installed on it: HP-UX in / etc/ exports: <m ount_point_of_CD> -ro,root=<m achine_nam e_1[:m achine2]> 17 Fognet’s Field Guide to OpenView NNM Solaris in / etc/ d fs/ d fstab share -F nfs -o root=<m achine_nam e> <m ount_point_of_CD> Red Hat Linux A dvanced Server 2.1 in / etc/ exports <m ount_point_of_CDROM> <m achine_nam e> (ro,all_squash) Installation issue (Window s): N on-standard directory N N M inserts inform ation abou t its install location into the registry. If N N M is be installed in a non-stand ard location (other than %System Drive%/ Program Files/ H P Op enView ), that su bd irectory shou ld be created p rior to installation. This w ill allow the nonstand ard location to be u sed to store d ata (rather than in %System Drive%/ Program Files/ H P Op enView / d ata). Create the <install location>/ d ata d irectory before installation. Installation issue (Window s): Terminal Services When installing N etw ork N od e Manager 7.5x u nd er Term inal Services, p roblem s m ight exist w here %System Root% d oes not correctly resolve in Com m and Prom p t w ind ow s. If this occu rs, a reboot is necessary. Unable to dow nload JRE through firew all Recent versions of N N M (7.01+) d o not com e bu nd led w ith JRE and au tom atically d etect if a com p atible version is installed on the server. If not, the install p rocess m ay attem p t to access su n.com to d ow nload the bits. Som e firew all ru les requ ire a resolvable DN S nam e for the d ow nload , bu t the Su n d ow nload site for JRE retu rns a virtu al host. Resolvable locations for JRE are available and can be fou nd in a w eb search. One su ch site is: http:/ / download .au.kd e.org/ pub/ java-sun If firew all ru les w on‟t even p erm it that, try to d ow nload and ru n the “Wind ow s Offline Installation” of the ru ntim e environm ent from : http:/ / java.sun.com / j2se/ 1.4.2/ dow nload.htm l D evices discovered in unmanaged state Objects that are beige in the m ap s are u nm anaged objects. Whenever a nod e or interface is d iscovered w hose IP ad d ress is in a su bnet that w as p reviou sly u nd iscovered by netmon, a new netw ork sym bol is p laced on the internet m ap in the u nm anaged state. This is to p reven t ru naw ay d iscovery. N ew netw orks need to be selected and then m anaged from w ithin the ovw m ap . 18 Common Issues When rand om d evices are d iscovered as u nm anaged , it cou ld be that the server‟s license lim it has been reached . Ru n the follow ing com m and to d eterm ine the nu m ber of d iscovered objects and licensed object cou nt: ovtopodump –l. Finally, d evices m ay be p laced on the m ap in the u nm anaged state d u e to the “-U” configu ration flag in the oid _to_typ e or H Poid 2typ e files. See p age 7 for m ore info on these files. Use a netmon seed file as a w ay to allow for the d iscovery of selected netw orks as m anaged . Also, scrip ts that search for u nm anaged d evices an d then m anage them externally can be w ritten to sim u late the N N M version 3.31 behavior of d iscovering the e ntire netw ork by d efau lt. See p age 41 for inform ation abou t m anaging objects externally and p age 42 for m ore abou t seed ing netmon. Cleaning orphaned objects from object database N N M d atabase irregu larities m ay m anifest them selves in several w ays. For exam p le, an error m essage ind icating that a p articu lar object cannot be ad d ed to the top ology u sing loadhosts is one ind ication the object exists in one or m ore N N M d atabases bu t has som ething w rong w ith it or is ou t of sync w ith another N N M d atabases. These sorts of issu es are also com m on in environm ents w here DN S is very d ynam ic or som etim es m isconfigu red . It is a good general p ractice to rep eat this p roced u re occasionally: Open all existing m aps and allow them to synchronize, close. ovstop netmon xnmsnmpconf –clearC ovw –mapcount –ruDv ovtopofix –a ovstart netmon Open all existing m aps and allow them to synchronize. D aemons ovet_da* not running (ovstatus) After enabling Extend ed Top ology, new d aem ons are registered throu gh the ARF object m anager. The ovstatus com m and m ay retu rn “N ot Ru nning” for d aem ons w ith nam es like ovet_d aDetails, ovet_d aCDP, ovet_d isco, etc. These d aem ons are su p p osed to be not ru nning m ost of the tim e, and m ight only d isp lay as ru nning w hen they are called to ru n d u ring an ET d iscovery cycle. 19 Fognet’s Field Guide to OpenView NNM They m ay also show as ru nning soon after an ovstart com m and is issu ed , bu t they m ay su bsequ ently show as not ru nning w hen and ovstatus com m and is issu ed . N etscape and N N M co-existence (UN IX) N etscap e m ight allocate the entire color m ap w hen started , w hich m ay w hack ou t ovw colored icons and cau se the error seen im m ed iately below . To force N etscap e to u se a p rivate color m ap , start it u p u sing: netscape –install Alternatively, the m axim u m nu m ber of colors, and/ or the u se of a p rivate color m ap can be sp ecified in the X Ap p Defau lts resou rce files for N etscap e. Cannot allocate 128 colors - using monochrome images On UN IX p latform s, a m essage sim ilar to above m ay occu r w hen there are not enou gh colors allocated to the colorm ap that N N M is requ esting to p rop erly d isp lay icons, etc. A sim ilar m essage m ay be: Allocation errors: 32 colors (128 requested ). To resolve, from w ithin CDE, bring u p the Style Manager. Und er Color, there is an op tion for “N u m ber of Colors.” There shou ld be an op tion u nd er that to set “More colors for ap p lications.” Also, see the section im m ed iately above and u se the “-install” N etscap e op tion, if u sing netscap e. If ru nning X over a virtu al w ind ow ing server su ch as Exceed or Reflections, be su re the server ap p lication is configu red to u se a color d ep th of 24. Unable to load any useable font set (UN IX) Sym p tom : When lau nching an xnm grap h w ind ow : Warning: Cannot convert string "-dt-interface system-medium-rnormal-m*-*-*-*-*-" to type FontSet Warning: Unable to load any useable fontset To resolve, ru n: xrdb -merge /usr/lib/X11/app-defaults/Xnmgraph 20 Common Issues xterm: unable to locate a suitable font (UN IX) Create a file in the hom e d irectory of the u ser called .Xd efau lts and inclu d e the follow ing line in the file: xterm*Font:-*-lucidatypewriter-bold-r-*-*-12-*-*-*-*-*-*-* Missing charsets in string to fontset (Solaris) Sym p tom : On lau nch of ovw, the follow ing errors retu rn: Warning: Cannot convert string "-dt-interface system-medium-rnormal-s*-*-*-*-*-" to type FontSet Warning: Missing charsets in String to FontSet conversion Warning: Unable to load any usable fontset… Resolu tion: 1. Startu p the font server on the Solaris w orkstation: / usr/ openw in/ bin/ fslsfonts -server <N N M Server>:7000 2. Enable the font server on the Solaris w orkstation: / usr/ openw in/ bin/ xset+fp tcp/ <N N M Server>:7000/ all Missing charsets in string to fontset (HP-UX) Sym p tom : On lau nch of ovw, the follow ing errors retu rn: Warning: Cannot convert string "-dt-interface system-medium-rnormal-s*-*-*-*-*-" to type FontSet Warning: Missing charsets in String to FontSet conversion Warning: Unable to load any usable fontset… Resolu tion: To tu rn on the H P-UX font server: 1. 2. 3. 4. 5. 6. 7. On the H P-UX server, ed it: / etc/ X11/ fs/ config Append the follow ing to the end of the "catalog=...." line: /usr/dt/config/xfonts/C Save the file and ed it: / etc/ rc.config.d / xfs Set the follow ing variable in the above file: RUN_X_FONT_SERVER=1 Save the file and run the follow ing comm and s: /sbin/init.d/xfs stop /sbin/init.d/xfs start On the target system enter: xset +fp tcp/<Server IP Addr>:7000 1>/dev/null xset fp rehash If xset is not in the path, append $XDIR environm ent variable. N TLM - passw ord needed - firew all: unknow n Often seen as an installation issue, this is a known issue with JAVA 1.4.2_01. To resolve, explicitly set the proxy server by selecting "Bypass 21 Fognet’s Field Guide to OpenView NNM proxy server for local addresses" dialog under LAN settings under the connections tabs in IE Internet Options when using a script for proxy detection. Similarly, if the local user account specified for anonymous IIS access is disabled due to site security policies, this needs to be changed. Troubleshooting daemons using ovstart and ovstatus To get m ore statu s for a d aem on: ovstatus –v <daemon> To get highest verbosity (UN IX only): ovstart –v –V –d <daemon> Automatically detect and restart OV daemons that die The follow ing log-only Op enView enterp rise event can be u sed to d etect and log failing d aem ons that are registered via ARF and “WellBehaved ”: OV_AppUngracefulExit .1.3.6.1.4.1.11.2.17.1.0.59179058 The fou rth varbind contains the nam e of the d aem on that exited u ngracefu lly, so in ad d ition to logging this event, set u p an au tom atic action that execu tes: ovstart $4 Filtering on custom ov w db fields The N N M Gu id e to Scalability and Distribu tion Ap p end ix A Table A -2 lists objects and attribu tes that can be u sed in a filter. The IPMAP_FILTER_FROM_OVW environm ent variable allow s filtering on cu stom ized field valu es and those ad d ed by third p arty p rod u cts. See p age 3 for inform ation on setting environm ent variables. Working w ith multiple JAVA versions Mu ltip le versions of JAVA m ay be installed on the N N M server or on any clients that are u sed to access the N N M server. See the su p p ort m atrix on p age 296 to d eterm ine w hich version of JAVA is requ ired for a p articu lar version of N N M. On the client sid e, the environm ent that invokes N N M need s to be configu red to p oint to the ap p rop riate version of JAVA. On UN IX system s set the environm ent variable $JAVA_H OME to p oint to the binary d irectory for the su p p orted JAVA installation, for exam p le: / op t/ java1.4/ bin. On Wind ow s system s, set the com p atible version via: Control Panel->Java Plu g-in. Often, other JAVA ap p lications requ ire d ifferent versions than those that are com p atible w ith N N M. Som e u sers resort to configu ring m u ltip le vend ors‟ 22 Common Issues brow sers w ith sep arately-configu red versions of JAVA. While certainly inconvenient, it solves som e cross-ap p lication issu es. It is ad visable w hen installing m u ltip le JAVA versions to install them increm entally from low est to highest version. N ote that Microsoft Internet Exp lorer brow sers su p p ort only a single installation onto any one system . Many other brow sers, how ever, can be installed m u ltip le tim es onto the sam e system , so each installed instance can be configu red to p oint to sep arate versio ns of JAVA. Client-sid e su p p ort for m u ltip le JAVA versions is im p roved w ith Version 1.5 and greater. The p rocess is to cop y the d irectories for the p re 1.5 versions from w here they w ere installed to a new location. Deinstall all p re 1.5 versions (Control Panel, Ad d / rem ove p rogram s) then install a 1.5+ version, w hich can be fou nd at: w w w .java.com / en/ d ow nload / w ind ow s_ie.jsp . Cop y/ m ove the p reviou s version d irectories to a su bfold er of the new ly installed 1.5+ version. Start Control Panel/ Java. Select Java/ View u nd er Java Ap p lication Ru ntim e Setings. Choose Find , and search and find the javaw.exe for the p reviou s version(s), Verify that the ap p lication d etects correct info u nd er Platform and Prod u ct. Select OK to exit Java Control Panel. 23 4. NNM and Name Services N N M relies heavily on looku p s for m ap p ing nam es to ad d resses and vice versa. The section on netmon d iscovery on p age 36 exp lains the im p ortance of stable and accessible nam e services to N N M. Using / etc/ hosts or LMhosts files based looku p s w orks w ell, bu t p rovid es very lim ited ability to u se nam ing services as a tool for associating m u ltip le interfaces w ith d evices w hen N N M cannot d o so otherw ise via SN MP. WIN S-based nam ing is su p p orted , bu t generally is not a best p ractice, since Microsoft itself is m oving aw ay from it in favor of DN S. If the WIN S server is not close to the N N M server, a netw ork ou tage cou ld rend er N N M p ractically u nu sable if it can‟t reach the WIN S server. The sam e is tru e of DN S servers, bu t DN S is au to m atically cached locally u nd er Wind ow s and caching DN S is often recom m end ed for UN IX N N M servers. DN S is the p referred nam e service for N N M. Other nam ing services have trou ble w ith the situ ation w here a rou ter cannot be ad d ressed by SN MP and N N M w ou ld otherw ise create sep arate nod es for each interface since it has no other w ay to know the interfaces belong to the sam e d evice. DN S can p rovid e the m ap p ing of m u ltip le interfaces to a single nod e nam e. H osts files can‟t d o this, nor can WIN S. How node objects are named Objects are assigned IP H ostnam es based on w hether netmon can find a nam e by d escend ing throu gh the follow ing checks: A non-m igratable uniquely-nam ed software loopback interface (not 127.0.0.*). Determ ined by querying ifType for softwareLoopback gethostbyaddr results on Preferred IP (see below ) SN MP sysN am e returned on Preferred IP (see below ) 24 NNM and Name Services Preferred IP: Low est num bered IP Add ress Low est Softw are loopback IP ad d ress (not 127.0.0.*) N AT ad d ress (introd uced in V6.4x) Low est non-m igratable IP Add ress Use the IPX server ad d ress (Window s only) Use "low est" IPX ad d ress (Window s only) LLa add ress ( MAC Ad d ress ) A m igratable nam e The current nam e (if present in DB) The NN M UUID The best p ractice is to assign softw are loop back ad d resses w herever p ossible. The ad d ress that resp ond s to SN MP and is the p referred ad d ress/ interface for m anagem ent, shou ld be given the A record in DN S, and a corresp ond ing PTR in its reverse zone. If other conventions are u sed , there m ay be a m ism atch in a nod e‟s nam e and the nam e u sed by N N M for trap s from that nod e. In all cases, reverse looku p s are heavily u sed by N N M so PTR record s shou ld alw ays be in p lace for all interfaces that N N M can‟t associate w ith a nod e throu gh SN MP. The PTR‟s for all other interfaces bu t the one u sed for m anagem ent, shou ld be a stand alone PTR p ointing t o the FQDN configu red in the A-record Assigning softw are loop backs has benefits other than for N N M alone. Using a loop back interface in OSPF can m ake OSPF netw orks m ore stable. Cisco in fact recom m end s u sing them . Other u ses for loop backs inclu d e load balancing betw een BGP p eers over tw o or m ore interfaces on Cisco rou ters. Cisco Works u ses m u ch the sam e ru les as N N M for selecting a p referred SN MP ad d ress SN MP sysnam e for any d evice can be w ritten from the N N M server if an SN MP w rite com m u nity is enabled for the d evice. This can be d one from the SN MP MIB brow ser or from the com m and line u sing: snmpset –c <writeString> <target> sysName.0 octetstring <nameString> N ote: Loop back ad d resses on rou ters in the sam e su bnet can cau se rou ting to not w ork p rop erly. This is avoid ed by u sing 32 bits m asks. 25 Fognet’s Field Guide to OpenView NNM If the rou ters su p p ort it and they are ru nning a classless rou ting p rotocol, a fu ll 32-bit m ask for the loop back IP can be u sed to save on IP ad d ress sp ace. This w orks fine w ith any version of N N M after 6.2, and Cisco rou ters w ith p ost 12.3 IOS. On p rior versions of N N M, the follow ing errors m ay be seen: From load hosts: "WARN IN G: Invalid broad cast IP Ad d ress 255.255.255.255, ignoring entry for <target>." From the GUI, the error is: "The IP ad d ress is the broad cast ad d ress for this netw ork." For m ore on this, see the section on 31 and 32-bit netm asks on p age 47. It is im p ortant to d raw the d istinction betw een nam es and labels in N N M. N am es are things that m u st be u niqu e in the object d atabase (ovwdb) and top ology (ovtop m d ) d atabase. In N N M, nod es have tw o kind s of nam es: IP H ostnam es and Selection N am es. Labels are the strings that show u p on the nod e in a m ap . How preferred IP address for SN MP is chosen The p referred SN MP ad d ress for a nod e is chosen accord ing to the above p roced u re. This is the ad d ress u sed to access the SN MP agent on the nod e. The p referred IP ad d ress u sed can be m anip u lated u sing the op tions listed on p age 28. How node objects are labeled N od e selection nam es are by d efau lt the sam e as the IP H ostnam e, w hich is d eterm ined accord ing to the ru les above. Users (and third p arty ap p lications) can change the selection nam e. If the selection nam e for tw o objects conflict, a nu m eric ID string is ap p end ed to one of the selection nam es in ord er to achieve u niqu eness. N od e labeling ru les, in d escend ing ord er of p reced ence, are: If the nod e has an IP hostname, truncate it to the basenam e (strip off d omain and subdomain nam es) If the nod e has a N etWare server nam e, use it, otherw ise, use the netw ork num ber of the internal server add ress If the nod e reports an SN MP sysNam e value, use it, truncating after any blanks in the returned value If the nod e supports IP, the IP address is form atted as string If the nod e supports IPX, the host-ad d ress portion is used If the nod e has a LLA/ MAC, th e physAd d r is form atted as a string 26 NNM and Name Services In environm ents w here the u se of SN MP sysN am e is p referred over any nam es assigned to the d evice in nam e services, u se the ip N oLooku p .conf, d escribed on p age 28. How interface objects are named by net mon In N N M 6.2 and later, if the agent su p p orts the ifAlias MIB variable and it is non-blank and u niqu e on the nod e, it shou ld be u sed , otherw ise a non-blank resp onse for ifN am e is u sed . If this isn‟t available, ifDescr w ill be u sed , bu t only the first p art u p to any blanks (u p to V7.51 Interm ed iate Patch 18, see section below ). To force the u se of ifN am e/ ifDescr only, u se the netmon LRF setting "-k u seIfAlias=false". ifAlias is fou nd in : ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifAlias If a d u p licate ifAlias is d etected for a nod e, N N M generates an alarm . Either resolve the d u p licate nam ing issu e (p referred solu tion) or force ifN am e globally w ith the netmon LRF sw itch m entioned above. See p age 5 for instru ctions on u sing the LRF p roced u re. When changing the ifAlias, or w hen changing the LRF sw itch, a d em and p oll to the nod e m ay or m ay not p rop erly u p d ate the object‟s label. There w as a know n p roblem w ith N N M 6.2 and ifAlias changes that w as ad d ressed via p atches. If it d oesn‟t u p d ate p rop erly, the nod e shou ld be d eleted from all m ap s and red iscovered . ifD escr naming restrictions N N M V7.51 Interm ed iate Patch 18 ad d ed su p p ort for w hite sp aces in the ifDescr SN MP variable. Prior to this, the text after the first w hite sp ace w as tru ncated . To allow the com p lete text of ifDescr to be u sed , set the follow ing environm ental variable to a valu e of “tru e:” OV_N N M_IFLABEL_IFDESCR How netw ork objects are named When a new netw ork object is d iscovered by netmon, the netw orks file or DN S is searched for m atching nam es. If non e are fou nd , the IP netw ork ad d ress for the netw ork is u sed . The files are: Wind ow s: UN IX: \ w indow s\ system32\ d rivers\ etc\ netw orks / etc/ netw orks 27 Fognet’s Field Guide to OpenView NNM To change the nam e of an existing netw ork nam e in N N M after u p d ating the netw orks file, ru n: ovtopofix –u <previous name for network object> N ode and interface naming restrictions The only allow able characters in a label for a host nam e in N N M are ASCII letters, d igits, and the d ash character. The u nd erscore character is illegal. In essence, N N M, like m any other ap p lications, enforces p reRFC 2181 ru les w ith resp ect to allow able characters in hostnam es. More recent versions of N N M m ay have relaxed som e of these restrictions. Labels m ay not be all nu m bers, bu t m ay have a lead ing d igit (e.g. 3com .com ). Labels m u st end and begin only w ith a letter or d igit, to a m axim u m of 63 characters. Letters are case-insensitive, thou gh p reN N M 5.0 versions p reserved case. N N M u ses this convention based on RFC1034 section 3.5. Less stringent nam ing conventions ap p ly to interfaces, how ever. Problem s m ay arise from the u se of certain characters in ifN am e and ifAlias entries, and interface nam e lengths are lim ited to 63 characters as w ell. ipN oLookup.conf, snmpnolookupconf, excludeip.conf ip N oLooku p .conf is a configu ration file w hich sp ecifies IP ad d resses against w hich looku p s w ill not be p erform ed . snmpnolookupconf is a com m and line u tility w hich m aintains the SN MP N o Looku p cache, w hich is p art of the SN MP configu ration d atabase (ovsnm p .conf). IP hostnam e entries in the N o Looku p cache w ill not have nam e service qu eries issu ed to them . If the above tools d o p rovid e the d esired resu lts, for exam p le, to ad d ress nam ing issu es stem m ing from the low est IP ad d ress being u sed , u se the exclu d eip .conf configu ration file to m anip u late the selection of the p referred IP ad d ress. General D N S considerations netmon d iscovers IP ad d resses first. To d eterm ine hostnam es, netmon p erform s reverse looku p s on the IP ad d resses u sing the gethostbyaddr system call. Reverse looku p s in DN S-configu red environm ents w ill fail u nless PTR record s are exp licitly d efined w ithin DN S. Failu re to 28 NNM and Name Services p op u late PTR record s in DN S is one of the m ost com m on cau ses of nam e service related issu es in N N M. To u se DN S to force a p articu lar ad d ress to be chosen over w hat N N M d efau lts to (often the nam e associated w ith the low est nu m bered IP ad d ress), u se the follow ing p roced u re. Create an ord inary A-record for the p rim ary interface w ith a corresp ond ing PTR-record in the reverse zone. For all other IP ad d resses, one can m ake another nam e record (A-record ) for the nod e, w ithou t a PTR p ointing back. Bu t for all those ad d resses, m ake a PTR p ointing at the A -record for the p rim ary nam e. Preferrably, in the PTR record s, take each of the nod e‟s IP Ad d resses and p oint them back to the com m on A -record . Then alias the other IP ad d resses in the forw ard looku p s w ith CN AME record s. If the reverse looku p d oes not m ap back to the A-record , then N N M m isses the connection back to the nod e w hen an SN MP trap d oes not com e from the A-record ‟s IP ad d ress. Cache-only D N S As m entioned above, there are several reasons w hy a cache-only DN S server shou ld be consid ered on the N N M server. The m ain reason is to p rovid e local control of d evice nam ing for netw ork m anagem ent p u rp oses. A local cache-only DN S u sed to be m ore frequ ently recom m end ed for N N M server p erform ance im p rovem ents, bu t H P m ad e som e red u ctions to N N M‟s u se of looku p s as w ell as ad d ed som e levels of u ser control over looku p behavior (see above). Still, a local DN S server w ou ld increase looku p sp eed and red u ce netw ork load , as N N M is still a heavy u ser of DN S. Another com p elling reason for a local DN S server is to elim inate the scenario w here the netw ork link betw een the N N M server and the DN S server are severed . N N M fu nctions are severely ham p ered by the loss of nam e services d u ring norm al op erations. N N M p rovid es som e caching of looku p d ata throu gh the SN MP configu ration cachedb, w hich is p art of the ovsnm p .conf d atabase. This d oesn‟t cache all looku p d ata, how ever, p articu larly d iscovery -related looku p s. N ote also that Wind ow s 2000 and above servers (and w orkstations) p rovid e a DN S client that acts as a cache-only DN S server, bu t this d oes not p rovid e any configu rability. It d oes bring into qu estion the reason to install a cache-only DN S server on N N M servers solely for 29 Fognet’s Field Guide to OpenView NNM p erform ance how ever. im p rovem ents or to p revent service d isru p tions, Split horizon D N S Sp lit H orizon DN S configu rations m ay offer greater flexibility and ease of control for netw ork m anagem ent p u rp oses over cache-only DN S. In general, there are three architectu res for sp lit horizon DN S: Tw o sep arate content DN S servers, each w ith d ifferent d atabases; a single content DN S server serving u p m u ltip le d atabases; and a single content DN S server serving u p a single d atabase w hose record s are tagged for visibility to sp ecific clients. With N N M, any of above op tions can be u sed to achieve finer control over netw ork m anagem ent nam ing w hile not com p rom ising netw ork ap p lications or secu rity. In fact, netw ork m anagem ent is often the p rim ary reason for u sing Sp lit H orizon DN S. Another com m on p ractice in this regard is to configu re a local DN S for a su bd om ain of d evices, w here that DN S server is then au thoratative for d evices only of interest to the netw ork m anagers and caching for all other d evices. BIN D -based D N S implementation troubleshooting The first versions of BIN D 9 broke the ability to coerce sortlist ord er for resou rce record sets that w as p ossible in BIN D 8 u sing the RRSET op tion show n below . Drop p ing RRSET m eant that rand om -cyclic behavior d eterm ined w hich DN S record w ou ld be retu rned first in the list, w hich is w hat N N M u ses to d eterm ine IP H ostnam e. The op tion w as re-instated in BIN D 9.2.3. The BIN D 8 and BIN D 9.2.3 (or later) m ethod for fixing retu rn ord er w ithin the op tions block in nam ed .conf is: rrset-order {class IN type A "hostA.local" order fixed;}; The follow ing nam ed .conf logging op tions are u sefu l for tracing w hat DN S qu eries are being generated by N N M: logging { channel "queryfile" { file "/var/log/dns-query.log" versions 4 size 5m; # 20MB rolling logs print-time yes; print-category yes; print-severity yes; severity info; }; category "queries" { "queryfile"; 30 NNM and Name Services }; }; BIN D -based D N S on Window s Microsoft‟s bu ilt-in caching “DN S Client” service m ay not p rovid e enou gh flexibility and control, as it is not at all u ser -configu rable. Microsoft‟s DN S Server is also d ifficu lt to w ork w ith (accord ing to som e). ISC BIN D is easy to install on the N N M server as a cache-only DN S server that p rovid es fu ll control, and is free. Be aw are, how ever, that BIN D, like SN MP, has been the su bject of secu rity vu lnerabilities in the p ast. More info can be fou nd on this throu gh SAN S. The follow ing step s ou tline configu ring ISC BIN D on Wind ow s: 1. 2. 3. 4. 5. 6. 7. 8. 9. Dow nload BIN D binary and d ocs from w ww .isc.org Extract and install package Run: rndc-confgen –a Create configuration files; see exam ples in d ocs or on w eb Check configuration files w ith namedcheckconf Start ISC BIN D service and monitor App Log events Test forward and reverse lookups w ith nslookup Disable Window s DN S Client Service Reboot and check event log again N ote: When m aking m ajor changes to nam e services that the N N M server su bscribes to, alw ays exp ect som e issu es w ith N N M‟s top ology and general stability. Often, it is ad visable to start d iscovery from scratch is su ch cases. See p age 12 for this p roced u re. “N ame services are performing poorly” alarms This OpenView enterprise event is generated when NNM’s internallygenerated lookups are taking too long or constantly timing out. If nodes are being polled via netmon, the following command may provide some information about lookup performance: ovstatus –v netmon See the appropriate section below to address performance issues with name services, and also refer to the NNM “Managing” guide in Appendix E and other sections. 31 Fognet’s Field Guide to OpenView NNM Tuning lookup performance (Window s) N N M u ses gethostbyname and gethostbyaddr system calls, bu t d efau lt N T settings for these requ ests can be very long. By d efau lt, N T shou ld p erform a N etBIOS looku p first, then a hosts file (\ w ind ow s\ system 32\ d rivers\ etc\ lm hosts) looku p . A N etBIOS looku p tim eou t is 4.5 second s, based on the follow ing registry valu es: H KLM\ System\ ControlSet001\ Services\ N etBt\ Param eters: N am eSrvQueryTim eout=1500 m sec N am eSrvQueryCount=3 If DN S or WIN S is configu red , typ ically these m ight be called on first, and then N etBIOS and hosts looku p s w ou ld be p erform ed . Make su re "Enable DN S for Wind ow s Resolu tion" is checked , and "Enable LMH osts Looku p " is u nchecked . The recom m end ed registry p aram eter settings are as follow s: H KLM\ System\ ControlSet001\ Services\ N etBt\ Param eters: Set Nam eSrvQueryCount to 0 Set Nam eSrvQueryTim eout to 100 to 500 Set Nod eType to 2 ( i.e. P-nod e ) N am eSrvQu eryTim eou t shou ld be set based on the overall latency in the netw ork. N ote that w ith these settings, access to other N T servers d oes not ap p ear to be affected , how ever access to LAN Manager servers d oes. Control the search ord er u sing the follow ing keys u nd er: H KLM\ System\ CurrentControlSet\ Services\ TcpIP\ ServiceProvider: DnsPriority LocalPriority H ostsPriority N etbtPriority These shou ld have valu es assigned to them the range of -32768 to 32767. The lower the nu m ber the higher the p riority. The p riority ord er d eterm ines the ord er that they are u sed . localPriority affects the nam e looku p u sing the lm hosts file. There is also another key: 32 NNM and Name Services H KLM\ System\ CurrentControlSet\ Services\ TcpIP\ Param eters\ DnsN btLookupOrd er This affects w hether DN S has p riority over N etBIOS. A valu e of 0 ind icates DN S has p riority, a valu e of 1 ind icates N etBIOS d oes. See MS d ocu m ent Q120642 for m ore inform ation. A best p ractice is to sim p ly d isable N etBIOS over TCP/ IP (N etBT). If this is d one, it shou ld only com p rom ise file sharing to/ from N T 4.0 servers. If not d isabling the N etBT, set the N N M server as an H -N od e or PN od e for N etBT resolu tion. This w ay, if a nam e isn't resolved by one of the other m ethod s, the server shou ld d o a qu ery d irectly to a WIN S server, w hich is m ore likely to su cceed on a large netw ork than a broad cast. A P-N od e d oes not d o a broad cast before giving u p , bu t is not allow ed on som e netw orks. This is set in: H KLM\ SYSTEM\ CurrentControlSet\ Services\ Tcpip\ ServiceProvid er w ith the key nam e being Class. A valu e of 8 m akes the com p u ter an H nod e. The Class valu e for B nod e is 1, for P nod e is 2, and for M nod e is 4. Keep in m ind that m ost Wind ow s DN S servers are configu red to d o a WIN S qu ery them selves if a na m e or ad d ress isn't in their zones. WIN S nam e resolu tion failu res can be m ad e to fail faster by ad ju sting: H KLM\ SYSTEM\ CurrentControlSet\ Services\ N etBT\ Param eters: BcastNam eQueryCount BcastQueryTim eout N am eSrvQueryRetries N am eSrvQueryTim eout . If Active Directory is involved Wind ow s u ses the DN S servers know n to Active Directory first, even if they are d ifferent then the configu red DN S servers. To sw itch to a locally-controlled DN S server, get the N MS Server ou t of the Active Directory tree and p oint it at a DN S Server that is not know n to Active Directory, p referably on the sam e su bnet. Active Directory is not requ ired to ru n N N M.. Tuning lookup performance (UN IX) The m ost im p ortant configu ration entry p oint for UN IX nam e services tu ning is the / etc/ nssw itch.conf file. This file sets the ord er and failover behavior for several services inclu d ing nam e services. The nslookup com m and op erates d ifferently on H P-UX and Solaris. On H P-UX, nslookup qu eries DN S, N IS, or hosts; on Solaris, nslookup 33 Fognet’s Field Guide to OpenView NNM qu eries DN S only. On Solaris, u se getent hosts <target> qu eries to test how looku p s are affected by nssw itch.conf. On H P-UX and Solaris, DN S tim eou ts can be tu ned d irectly in the / etc/ resolv.conf file. By d efau lt, the tim eou t is calcu lated u sing the RES_RETRAN S p aram eter w hose d efau lt is 5000 m illisecond s (H P -UX) and 5 second s (Solaris) and RES_RETRY w hose d efau lt is 4. The follow ing entries lim it the tim eou t 1s p er failed looku p : domain <domain> nameserver <primaryNameServer> nameserver <secondaryNameServer> retrans 1000 # value in milliseconds retry 1 N N M doesn’t detect D N S changes (Window s) With Wind ow s 2000, Microsoft introd u ced a local caching -only DN S server that gets qu eried first for any DN S looku p s, regard less of looku p ord er. This is the DN S Client service, and it is installed and ru ns au tom atically by d efau lt. Od d valu es cou ld still be retu rned from the local client after another DN S server is u p d ated . Disabling the DN S Client service if ru nning a caching-only DN S on the N N M server, is recom m end ed on the N N M server as N N M alread y caches qu eries via the SN MP Configu ration d atabase. That cache can be cleared u sing: xnmsnmpconf -clearC To d u m p and then clear the contents of th e local DN S cache , ru n: ipconfig /displaydns ipconfig /flushdns To A record or not to A record N ew er versions of N N M have m ore intelligence and flexibility for p icking the p referred m anagem ent IP ad d ress (aka p rim ary IP ad d ress). For p rim ary IP Ad d resses, alw ays ad d an A record and a PTR record for the resverse zone. Ad d ing A record s for second ary IP ad d resses is generally not necessary for N N M‟s p u rp oses, bu t m ay be d esirable for other reasons. When ad d ing A record s for second ary IP ad d resses, instead of ad d ing a PTR record p ointing back to that A record ‟s nam e, u se the nam e associated w ith the A record for the p rim ary IP ad d ress. 34 NNM and Name Services A record compare issue on Solaris platform Solaris im p lem ents sp ecial logic in gethostbyaddr su ch that after the OS requ ests the PTR from the DN S and gets the A record back, it issu es a su bsequ ent requ est for the A record and com p ares the resu lts to the IP ad d ress originally requ ested . If this ad d itional requ est isn't retu rned , the w hole call retu rns N ULL. This logic w as institu ted as IP sp oofing p rotection. This can be p roblem atic on m u lti-hom ed hosts that d o not have A record s for every IP ad d ress d efined for the nod e in the case w here A record s cannot otherw ise be entered for all interfaces in DN S d u e to issu es w ith rou nd robin, ap p lication requ irem ents, etc. Work -arou nd s inclu d e only entering the loop back ad d ress for the d evice in DN S or u sing a com bination of noIp Looku p .conf and snmpnolookupconf. See p age 28 for m ore on these settings. nscd issues on Solaris The N am e Server Cache Daem on, nscd, is intend ed to sp eed u p nam e service looku p s, w hether from N IS, N IS+, DN S, or local files. In som e cases, how ever, nscd can slow DN S looku p s, and som e ad m inistrators choose to d isable it. Moreover, nscd interferes w ith rou nd robin (nscd caches record s in one ord er and d oesn't rotate them ). Som e im p ortant / etc/ ncsd .conf file tu nables are: Toggle host lookup caching: TTL positive results: TTL negative results: enable-cache hosts <yes | no> positive-time-to-live hosts <sec> negative-time-to-live hosts <sec> N ame resolution support commands: Use the follow ing com m and s to d eterm ine if N N M is p erform ing p oorly w ith resp ect to nam e services: $OV_SUPPORT/gethost <hostname> $OV_SUPPORT/gethost –v <host> (fw d and reverse lookup) $OV_SUPPORT/gethost –a <IP Address> xnmsnmpconf –resolve <target> xnmsnmpconf –dumpcache all xnmsnmpconf –clearC snmplookupconf -dumpC snmplookupconf –t[est] <target> $OV_SUPPORT/checkDNS.ovpl ovstatus –v snmpCollect ovstatus –v netmon 35 Fognet’s Field Guide to OpenView NNM N N M D N S tracing Set the follow ing environm ent variable to enable DN S tracing to the sp ecified file, then restart the Op enView d aem ons: OV_NS_LOG_TRACE=<directory>;0.0;2; See p age 3 for inform ation on setting environm ent variables. Logfile entries have the follow ing form : N:2:N:... N:2:A:... H ostnam e to IP Ad d ress lookup IP Ad d ress to H ostnam e lookup N ame services-related documentation N N M 6.2 p atches introd u ced DN S p erform ance im p rovem ents w hich are d iscu ssed in the follow ing w hite p ap ers: H P‟s White Paper: $OV_DOC/ WhitePapers/ DN Sand OpenView .pd f D. Stevenson‟s DN S White Paper: w w w.fognet.com / d s-d ns.doc N N M in demo, D HCP or mobile environments From an SNMP management perspective, the NNM server needs to be a fixed IP address that doesn‟t change often. Changing the IP address of the NNM server causes SNMP devices to generate traps to all the addresses that NNM discovers them from, because the NNM server sets itself as a trap destination when it can. If NNM is to be installed on a laptop that is moved around, or it is a demo system where the IP address is expected to change, set the USE_LOOPBACK=ON parameter in the following file to keep NNM daemons from crashing when the network becomes unavailable: $OV_CONF/ov.conf. Typically, in this scenario, the NNM license is instant-on, otherwise it would report an error if the licenced IP changed. 36 5. Discovery via netmon N N M has tw o d iscovery engines and tw o p olling engines, each of w hich ad d ress OSI level 2 and level 3 resp ectively, and each of w hich has overlap onto the other layer. netmon is the level 3 d iscovery and p olling engine and is a legacy p rod u ct elem ent. ovet_disco and ovet_poll are the new er level 2-focu sed d iscovery and p olling d aem ons. The ET p ollers d ep end on netmon for d iscovery, and the ovet_bridge p rocess is resp onsible for keep ing the sep arate top ology d atabases synchronized . Distribu ted N N M architu re is bu ild m ostly on legacy level 3 top ology and d iscovery, and ET d iscovery is not necessarily easily com p atible w ith d istribu ted installations. When N N M is first installed , netmon d iscovery occu rs au tom atically u nless d isabled d u ring d ialog at installation. To get layer2 top ology d isovered , the ovet_disco-based d iscovery m u st be enabled u sing the follow ing com m and : setupExtTopo.ovpl Sim ilarly, the ovet_poll-based p oller and the ad vanced p roblem analyzer , if d esired , m u st be enabled u sing the follow ing com m and : ovet_apaConfig.ovpl –enable APAPolling Prior to ru nning ovet_disco-based d iscovery, at least som e of the environm ent shou ld be p rop erly d iscovered by netmon first. Sim ilarly, p rior to enabling the ovet_poll-based p oller, the ovet_disco-based d iscovery m u st be ru n first. All netmon p olling op tions that can be set in the Op tions-N etw ork Polling Configu ration m enu s can be controlled from the com m and line as w ell u sing the xnm p olling com m and . For exam p le, to toggle netmon d iscovery on or off, u se: xnmpolling -ipDiscoveryOff or xnmpolling -ipDiscoveryOn 37 Fognet’s Field Guide to OpenView NNM Preparing for N N M discovery The H P d ocu m entation covers this w ell, bu t here is a review of som e im p ortant step s to take before setting N N M loose on the netw ork: 1. 2. 3. 4. Make sure SN MP com m unity strings are properly configured Make sure nam e services are OK. Read the section on nam ing Populate netmon.noDiscover file w ith und esirable d evices Populate netmon.noDiscover file w ith H SRP d evices (skip if running N N M 7.? or above, or use netmon.m igratable) 5. Have any ed ge routers? Consid er –R netmon LRF sw itch 6. Make sure firew alls are open to desired areas (see page 283) 7. Check for cut SN MP tables on key d evices 8. Set loopback ad d resses on routers, m ap to DN S (see page 24) 9. Run: nmcheckconf (UNIX) to probe network configuration 10. If there are nod es w ith many interfaces that are not d esirable to d iscover/ manage, consid er using netm on.interfaceNoDiscover After N N M discovery After initial au tom atic d iscovery by N N M, p erform these step s to e xp and the d iscovered environm ent: 1. 2. 3. 4. 5. Select d esired unmanaged n etw orks (beige) and select Map - Manage Objects to d iscover into ad jacent netw orks Run the follow ing and com pare to entries in oid_to_type, oid_to_sym , and H POid2Type (com mand is case sensitive): ovobjprint –a ―SNMP sysObjectID‖ See the script on page 45 for a way to automate the above step Check totals on objects d iscovered by running: ovtopodump -l Once ET Discovery is complete, compare ovtopodump –l output with ovet_topodump -info output D iscovery process in a nutshell netmon qu eries the SN MP MIB of the N N M Server to d eterm ine its IP ad d ress, su bnet m ask, its d efau lt rou ter's IP ad d ress and its local ARP cache. netmon read s the ARP cache of the d efau lt rou ter via SN MP, then send s an ICMP ping to each IP ad d ress learned from the ARP cache. See p age 49 for a list of the sp ecific MIB variables qu eried for this inform ation. Those nod es that resp ond to SN MP get an SN MP requ est for sysObjectId for each d iscovered nod e, and a series of other qu eries (listed below ) to d eterm ine other attribu tes an d cap abilities. To 38 Discovery d eterm ine the nod e typ e, N N M m ap s the agent OID to the oid _to_typ e or H Poid 2typ e file for vend or, SN MP agent and other top ology flags that d eterm ine w here in the m ap the nod e object shou ld be p laced . The oid _to_sym is then qu eried to m ap a sym bol to the nod e. Ad d itional nam e service qu eries are m ad e and the nod e is assigned a selection nam e accord ing to the ru les on p age 24. D iscovery polls vs. other types of net mon polls Discovery p olls focu s on find ing new nod es that haven‟t been fou nd before. The d iscovery p oll looks at the rou ting table, etc., and if it find s som ething new , netmon issu es su bsequ ent cap ability and top ology checks, otherw ise it d oesn't. A configu ration check or configu ration p oll com bines a cap ability p oll and a top ology check, this com bo being the equ ivalent of issu ing the nm d em and p oll com m and . The cap ability p oll looks at object DB cap abilities. The top ology p oll looks at top ology MIB tables su ch as the brid ge MIB, etc. The top ology p oll is only issu ed to those to connector nod es, and can be configu red to occu r m ore frequ ently in the p olling op tions. Understanding net mon-based level 2 discovery As m entioned above, there are som e vagaries in this legacy p oller. The sw itches and op tions that affect netmon’s d iscovery, p olling, and su bsequ ent top ology p lacem ent in IPMAP are nu m erou s. For the m ost p art, it is best to leave N N M‟s L2 settings alone. For the brave, the level2conf w hite p ap er exp lains m u ch of the aforem entioned vagaries. Becau se it typ ically raises m o re qu estions than it answ ers, this w hite p ap er w as rem oved from later d istribu tions of N N M. In fact, it has been p u rged from all of H P‟s p u blic p ages, bu t it can still be fou nd at several sites via w eb search . Limiting discovery $OV_CON F/ oid _to_typ e has global op tions to lim it d iscovery, for exam p le, of all non-SN MP su p p orted d evices. All system s m atching OID‟s p laced in that file can also be configu red to be ignored by N N M d iscovery. Details on u sing oid _to_typ e can be fou nd on p age 7. To d isable au tom atic d iscovery, toggle the “Discover N ew IP N od es” rad io bu tton in the IP Discovery configu ration area of the Op tions- 39 Fognet’s Field Guide to OpenView NNM N etw ork Polling Configu ration m enu bar item . The sam e can be accom p lished from the com m and line w ith: xnmpolling -ipDiscoveryOff To lim it d iscovery by IP ad d ress range, IP ad d ress w ild card , or by MAC ad d resses, see the m an/ ref p ages for netm on.noDiscover and netm on.noMACd iscover. Any changes to these files requ ire that netmon be restarted u sing ovstop and ovstart and only ap p ly to new nod es being d iscovered . To lim it d iscovery of interfaces based on ifTyp e, nam e, d escrip tion etc, u se netm on.interfaceN oDiscover (available in N N M 7.5+). This w ill p revent the interfaces from being d iscovered both by netm on and ET d isco. See Page 53 for m ore inform ation on this. Lim iting exp ort of nod es from netmon to ET is configu red in the brid ge.noDiscover file in $OV_CON F. See the brid ge.noDiscover m an/ ref p ages for m ore inform ation on this. With the release of N N M 6.31, H P introd u ced the ability of filters to be u sed to p revent d iscovery of w hole classes of nod es based on stand ard N N M filters. A d iscovery filter u ses N N M filter d efinition langu age (see p age 268) to d iscard or inclu d e only those nod es w hich p ass the filter. For exam p le, to d iscover only nod es w ith Cisco and Extrem e SN MP agents, bu ild a filter like this: DiscoInclud e “Cisco, Extrem e” { (“SN MP sysObjectID” ~ “1.3.6.1.4.1.9.*”) | | (“SN MP sysObjectID” ~ “1.3.6.1.4.1.1916.*”) } Or to exclu d e Microsoft nod es from d iscovery: DiscoExclud e “Microsoft: { { isN od e && ( vendor != "Microsoft" ) } Ru n ovfiltertest to see the resu lts of the filter if any of the d evices to inclu d e or exclu d e have alread y been d iscovered . To activate the d iscovery filters, u se the m enu bar p u ll d ow n: Op tions:N etw ork Polling Configu ration:IP. Select Configu ration Area: 40 Discovery General. Select Use Discovery Filter, then enter the filter nam e , for exam p le, DiscoExclu d e. The follow ing external com m and s can be u sed to d ep loy and control a d iscovery filter nam ed <filter-name>: xnmpolling –discFiltName <filter-name> -discFiltOn xnmpolling –discFiltName <filter-name> -discFiltOff Once the filter is configu red , tested , and ad d ed to the p olling configu ration, all nod es to be exclu d ed m u st be d eleted from the top ology and all existing m ap s m u st be op ened . Once they are d eleted , they w ill not be red iscovered . Externally managing or unmanaging objects Objects are typ ically m anaged or u nm anaged u sing m ap op erations in the GUI, bu t here are fou r com m and s u sed to m anage or u nm anage objects externally (in increasing ord er of com p lexity): xnmtopoconf ovtopofix topology URLs ovautoifmgr See p age 233 for inform ation on ad d ing, d eleting, m anaging or u nm anaging objects via top ology URLs. N ote that none of the above tools can be u sed to control d iscovery by ET or p olling by APA. In N N M 7.5 and above, u se ovet_toposet.ovpl to u nm anage nod es, board s and interfaces from being statu s p olled by the APA. More inform ation on ovet_toposet.ovpl can be fou nd on p age 75. xnmtopoconf can be u sed to m anage or u nm anage ind ivid u al nod es from the com m and line. xnmtopoconf is only available on N N M ed itions su p p orting DIM (i.e. not N N M Stand ard Ed ition). Use ovtopofix –G/g to m anage/ u nm anage objects by OOID. For exam p le, get OOIDs for all interfaces w hose nam e is Se0/ 1.1, then p ass to ovtopofix –G: ovobjprint -a "SNMP ifName" "SNMP ifName"="Se0/1.1" The follow ing tw o com m and s, resp ectively u nm anage (-g) and m anage (-G) objects: ovtopofix –g <OOID> ovtopofix –G <OOID> 41 Fognet’s Field Guide to OpenView NNM Use ovautoifmgr to au tom atically m anage or u nm anage interfaces based on filters. For exam p le, an ad m inistrator m ay w ant to m anage only interfaces w ith IP ad d resses assigned on sw itches. First, set u p tw o filters for the ovau toifm gr.conf file, one that p assed the nod es w hose interfaces are to be scanned by the second filter. In the filters file, d efine tw o filters: Switches "Switches Node Filter" { isBridge } NonIpInt "Non-IP Interfaces filter" { !isIP } Then, in the ovau toifm gr.conf file, ad d : Switches NonIpInt Stop discovery of non-IP interfaces for a device class Use the –N flag in the oid _to_typ e file. Changes affect only new ly d iscovered d evices, or ru n : ovtopofix -o <OID>. Details on u sing oid _to_typ e can be fou nd on p age 7. Stop all SN MP discovery/polling to a device class On som e d evices, it is not d esirable to have SN MP visibility, bu t sim p le ICMP statu s to the nod e is OK. Use the –I flag in the oid _to_typ e file to force ICMP-only statu s. Changes affect only new lyd iscovered d evices, or ru n : ovtopofix -o <OID>. Details on u sing oid _to_typ e can be fou nd on p age 7. D iscovery hints A com m and line op tion to p rovid e netmon d iscovery hints w as ad d ed in interm ed iate p atch 18 to V7.51. This allow s hints to be entered d irectly from the com m and line and can red u ce or illim inate the need to u se seed ing (see below ) and is a m ore d irect m ethod that u sing the loadhosts com m and (p age 45). To ad d a hint to netmon, issu e the follow ing com m and : netmon –h <ip addr> Seeding discovery Use a netmon seed file in environm ents w here red iscovery of the netw ork is frequ ent and d evices cannot be d iscovered by netmon d u e to netw ork or SN MP access. Entries in the seed file shou ld be p op u lated into p reviou sly-u nd iscovered netw orks as m anaged netw orks. This can save tim e w here otherw ise, new netw orks m u st be m anu ally m anaged from the GUI. 42 Discovery A seed file m ight also sp eed the d iscovery p rocess. A seed file has the form of a stand ard hosts file and is configu red u sing the –s <seedfile> LRF sw itch to netm on.lrf. See p age 5 for LRF instru ctions. N ote w hen sp ecifying a file in the sw itch for Wind ow sbased servers, the colon m u st be escap ed , for exam p le: -P -s "c\:\seedfile.txt" –k segRedux=true Configuring multiple SN MP community strings The netm on.cm str file su p p orts an ord ered list of com m u nities to u se w hen contacting SN MP agents for read -only access. H ere is an exam p le for a list of five com m u nities to attem p t to u se in ascend ing ord er to contact all SN MP agents: "cs1", "cs2", "cs3", "cs4", "cs5" : : : : Preferred SN MP management address If a configu ration p oll fails to reach a nod e‟s p referred SN MP ad d ress as set in the SN MP configu ration DB, netmon issu es a sp ecial “p ickSnm p Ad d rPoll” that attem p ts to find a valid alternative SN MP p referred ad d ress. This can cau se issu es w ith d ata collections and cou ld cau se the target‟s nod e nam e to change in N N M. If loop backs are not w id ely im p lem ented , p ickSnm p Ad d rPoll is a u sefu l tool to help m anage changing SN MP m anagem ent ad d resses. Also, see p age 26 and su bsequ ent sections for tip s on u sing DN S to u nd erstand the best netw ork nam ing conventions to m ost efficiently SN MP m anage netw orks. Defining loop back ad d resses on netw ork d evices also m ay obviate the need to w orry abou t the p referred SN MP m anagem ent ad d ress changing, since m ost netw ork d evices w ill re-assign the configu red loop back to an active interface if the cu rrent interface fails. If this is the case, follow the step s below to d isable p referred m anagem ent ad d ress p icking w hich is enabled by d efau lt. To d isable the p ickSnm p Ad d rPoll featu re, set the follow ing LRF (see p age 5) sw itch in netmon: -k pickSnmpAddrPolls=false Another –k netm on lrf op tion w as ad d ed in N N M 7.53 that shou ld also be d isabled to stop netm on from sw itching p referred m anagem ent ad d resses. The d ifference betw een this new er one and 43 Fognet’s Field Guide to OpenView NNM p ickSnm p Ad d rPolls is that the above op tion affects general SN MP p olls w hile the new op tion affects netmon-based SN MP statu s p olls: -k adjustNodeSnmpAddr=false Forcing the SN MP address for a node Use this m ethod if all other m echanism s for p icking the correct p referred SN MP ad d ress fail. If u sing the APA, see p age 92 for how ET and the APA can affect p reffered m anagem ent SN MP ad d ress. To force N N M to change the SN MP Ad d ress to a d ifferent one, follow these step s: 1. If using APA, d isable polling using: ovet_apaConfig.ovpl –disable APAPolling 2. In the IPMAP top ology map, tem porarily unm anage all the interfaces for the target nod e except the interface that is to be set as the SN MP Add ress 3. Demand Poll the d evice 4. Unless the netmon –k pickSnmpAddrPolls w as set to false, the preferred SN MP ad d ress w ill now be set to the managed interface. 5. Run: ovtopodump –l |grep ―SNMP ADDRESS‖ 6. Re-m anage the rest of the nod e‟s interfaces 7. Re-enable APA using: ovet_apaConfig.ovpl –enable APAPolling If there is the need to p erform this task for m ore than a hand fu l of nod es, a com m and line tool called topodbpoke can be u sed to change the p referred SN MP Ad d ress in the ovw top ology. topodbpoke w as ship p ed w ith N N M 7.53 in the $OV_SUPPORT d irectory. Instu ctions for u sing it can be fou nd in the follow in g d ocu m ent u nd er “Change Incorrect IPv4 Managem ent Ad d ress” in this w hite p ap er: $OV_DOC/ w hitepapers/ ETand APADeploym entGuid e.pd f If ru nning V7.53, this d ocu m entation can also be fou nd in the Gu id e to Using Extend ed Top ology u ser m anu al. Before N N M 7.53, the tool w as available throu gh H P su p p ort. The au thor had a need for this tool in October 2007 and op ened a case w ith H P, bu t H P su p p ort w as u nable to locate the u tility. For this reason, and for folks ru nning earlier versions of N N M, the au thor has p osted the topodbpoke cod e for all p latform s excep t Linu x here: w w w.fognet.com / topod bpoke.zip 44 Discovery net mon’s auto-adjusting discovery polling algorithm Discovery p olls are basically SN MP requ ests to rou ting table s to find nod es that haven‟t been d iscovered before. All nod es start at 15 m inu te intervals for new nod e p olling. If netmon d iscovers no new nod es d u ring a p oll, then five m inu tes are ad d ed to the p olling interval. If m ore than 5 nod es are d iscovered d u rin g a p oll, the p olling interval is halved . When a new nod e is d iscovered , a configu ration p oll is issu ed . Su bsequ ent configu ration p olls are issu ed every 24 hou rs by d efau lt. PERL script: find OID s not listed in oid_to_type This scrip t p rod u ces a list of SN MP OIDs that have been d iscovered by netmon, bu t d o not have corresp ond ing entries in the oid _to_typ e or H Poid 2typ e files. Consid er ad d ing the resu lting OIDs to the files and setting d esired top ology flags for their treatm ent. N ote that on Wind ow s, the resu lts p rod u ced are not m ad e u niqu e. #!/usr/local/bin/perl open IN1,"c:\openview\conf\oid_to_type"; open IN2,"c:\openview\conf\HPoid2type"; @oids1 = <IN1>; @oids2 = <IN2>; close IN1; close IN2; @oids = (@oids1,@oids2); open CMD,qq(ovobjprint -a "SNMP sysObjectID" |); @objdb = <CMD>; close CMD; foreach (@objdb) { s/.*\"\.1.3.6.(.+)\"/1.3.6.$1/ or next; chomp; next if $oidhash{"$_"}; $oidhash{"$_"} = 1; my @found; foreach $x (@oids) { push @found, $x if $x =~ /^$_/; } print "$_ not in oid_to_type\n" unless @found; } loadhosts The loadhosts com m and is u sefu l for ad d ing nod es to the N N M top ology ou tsid e of the d iscovery p rocess, bu t it can p rod u ce u nintend ed resu lts. If ad d ing a nod e into a netw ork t hat d oes not yet exist in the m ap , be p articu larly carefu l, since N N M m ay m ake certain assu m p tions abou t the “classfu llness” of the netw ork. In the p ast, loadhosts w as the p referred m ethod for forcing d iscovery for a nod e or set of nod es that netmon cou ld n‟t otherw ise d iscover. Since the release of Interm ed iate Patch 18 to V7.51, the best p ractice to accom p lish this is to give netmon d iscovery hints u sing the “-h” op tion to netmon. See p age 42 for d etails. 45 Fognet’s Field Guide to OpenView NNM To m ake su re loadhosts honors the “classfu lness” of the netw ork of the d evice being ad d ed . Use the “–m‖ op tion to sp ecify the netm ask for the netw ork the d evice lies in if it is in a netw ork not alread y p resent in N N M top ology. Still, loadhosts m ay com p lain, so it m ay be necessary to m anu ally create a netw ork sym bol in ovw w ith the correct m ask and then m anu ally ad d the d evice u sing m ap op erations. To d o this, see p age 196. If ad d ing less than 100 ad d resses, rem em ber t o u se the –n 1 sw itch. Page 233 p rovid es inform ation on ad d ing, d eleting, m anaging or u nm anaging objects via top ology URLs. This facility w as d esigned to p rovid e a secu re rem ote front end for loadhosts. loadhosts is d esigned to take inp u t from a file, bu t w hen invoked from the com m and line, shou ld take stand ard inp u t. This m eans that after inp u t is entered for the loadhosts com m and , the p rom p t m ight hang u ntil it is interru p ted w ith a “Ctrl C” com m and . loadhosts has m any com m and line op tions, bu t the best p ractice is to load a host w ith m inim al sw itches and let netmon d iscover the rest. For connectors, it is best to u se the ad d ress associated w ith the softw are loop back to seed netmon, bu t in som e environm ents, it is better to u se the ad d ress associated w ith the DN S nam e assigned to the d evice. N am e services m ay be u sed by netmon to d eterm ine w hat interfaces are associated w ith a p articu lar host if it can‟t otherw ise d eterm ine this throu gh SN MP. loadhosts byp asses netmon‟s norm al d iscovery sequ ence, so u p on load ing, a new nod e m ay not receive a fu ll configu ration check. This m eans that the icon m ay not be p rop erly rep resented on the m ap . Force the configu ration p oll u sing the GUI, nmdemandpoll, or u se the “-o” op tion to loadhosts to force an oid _to_typ e looku p if the d evice‟s SN MP agent OID is listed there. loadhosts w ill not su cceed in entering a p articu lar nod e into the top ology if the netw ork it is load ed into is in the u nm an aged state. Batch load hosts having a su bnet m ask of 255.255.255.248, w here the file contains an / etc/ hosts-like list of IP ad d resses and hostnam es: loadhosts –bvpm 255.255.255.248 <filename> To ad d a single nod e via p ip ing (w orks on Wind ow s, too): echo ―192.168.1.9 patchy‖ | loadhosts –V –v –m 255.255.255.0 46 Discovery loadhosts op erations m ay fail for a variety of reasons. A typ ical error m essage m ight say that the su bnet m ask is invalid . The m ost com m on cau se is that there is an incorrectly classed netw ork d efined (see p age 203). There are other reasons loadhosts m ay fail, su ch as firew alls, etc. In this case, ad d the objects m anu ally u sing the p roced u re on p age 196. In general, if the object can be reached via ICMP or SN MP, it can be load ed into the top ology. Objects that can‟t be reached by either p rotocol m ay be ad d ed , bu t can never be au tom atically connected in the top ology or given a statu s by netmon or the APA. D iscovery of 31 and 32 bit subnetw orks Both 31 and 32 bit netw orks u se the netw ork ad d ress as an IP Ad d ress. These are IP netw orks w ith a su bnet m asks of, resp ectively, 255.255.255.254 and 255.255.255.255. The N N M top ology m anager has trou ble w ith these, since it u ses the netw ork ad d ress to n am e netw ork level objects. To p rop erly hand le this situ ation, the -1 and -2 sw itches w ere ad d ed as LRF sw itches to the ovtop m d d aem on. These sw itches get arou nd the nam ing p roblem by ap p end ing / 31 or / 32 (as ap p rop riate) to the netw ork object nam e in the object d atabase. See the ovtop m d m an/ ref p age for m ore d etails and see p age 5 for inform ation on m od ifying LRF files. D iscovery of HSRP, clusters, and multilinked routers Legacy N N M hand les m igrating or d u p licate IP ad d resses very p oorly . N N M requ ires forcing interfaces that share the sam e IP ad d ress to go u nd iscovered by N N M‟s IPMAP top ology by p lacing them in the netm on.noDiscover file. Alternatively, the d evices that ru n H SRP can be configu red w ith “D” flag in the oid _to_typ e file, w hich assu m es that if the floating ad d ress is not in the ip Ad d rTable, then the ad d ress shou ld go u nd iscovered . Unfortu nately, Cisco started rep orting floating ad d resses in the active rou ter‟s ip Ad d rTable w ith IOS 12.1(14) and greater and 12.2(13) and greater, so the netm on.noDiscover file is the best bet . In N N M 7.5 the follow ing tw o LRF flags w ere ad d ed to the netmon d aem on: -k m igrateH srpVirtualIP=true -k d oN otDiscoverDuplicates=true 47 Fognet’s Field Guide to OpenView NNM The first allow s the au tom atic p op u lation of virtu al grou p IP ad d resses into the netm on.m igratable file d u ring d iscovery. The second allow s d u p licate IP ad d resses to be p op u lated into the netm on.noDiscover file d u ring d iscovery. Both are set to false by d efau lt. These settings, once ap p lied , w ill only affect new ly -d iscovered d evices. N ote that there w as a bu g w ith m igrateH srp Virtu alIP that w as ad d ressed w ith an early p atch to V7.5.In som e cases, the u se of the netm on.m igratable file can be u sed for clu stered nod es. H SRP virtu al grou p IP ad d resses can also be p laced in that file. Cisco‟s im p lem entation of m u ltilink PPP and Ju nip er‟s Mu lti-Rou ter Au tom atic Protection Sw itch ing featu re both create active interfaces that share a single ad d ress. In essence, netmon cannot accom m od ate the d u p licate interface ad d ressing betw een the rou ter p airs involved becau se both interfaces are active at the sam e tim e. Contrast this to an H SRP-like situ ation w here a single virtu al ad d ress is only active in one p lace at one tim e. These are best m anaged u sing the netm on.m igratable file. See the netmon m an/ ref p age for m ore inform ation and see p age 5 for how to u p d ate LRF flags. See the p roced u re on p age 7 for configu ring oid _to_typ e. See p age 243 for d etails on how ET hand les VRRP. See p age 245 for m ore info on how ET hand les H SRP. See p age 281 for d etails on m anaging m igratable IP ad d resses. Summary of net mon configuration files The follow ing external files influ ence netmon’s behavior, bu t the m ain settings for netmon are controlled throu gh the xnmpolling and xnmsnmpconf interfaces available from the Op tions m enu of ovw: $OV_CON F/ polling $OV_LOG/ netmon.trace $OV_CON F/ oid _to_type $OV_CON F/ H Poid2type $OV_CON F/ exclud eip.conf $OV_CON F/ netm on.noDiscover $OV_CON F/ netm on.MACnoDiscover $OV_CON F/ netm on.interfaceNoDiscover $OV_CON F/ netm on.m igratable $OV_CON F/ netm on.cm str $OV_CON F/ netm on.equivPorts $OV_CON F/ netm on.snm pStatus $OV_CON F/ netm on.statusIntervals $OV_CON F/ netm on.statusMapping $OV_CON F/ nm d emand poll.ports 48 Discovery Undocumented net mon LRF sw itch settings See p age 5 for d etails on the LRF p roced u re. -A Automatically m anage netw orks w hen created . -H 0 Don't do http d iscovery ( UN IX only ) -H 1 Do http d iscovery ( UNIX only ) -2 Don't use SN MPv2C -k configPollDelay <#> (see below) Im p lem enting the –H 0 and –H 1 sw itches im p roves the p erform ance and scalability of d iscovery p olls and of configu ratio n p olls. The equ ivalent for Wind ow s-based N N M servers is to ad d the –nohttp and -nohttpmg sw itches to the ovcap sd .lrf file. The H TTP d iscovery sim p ly sets a cap ability flag if the target resp ond s to a p oll to p ort 80. The only p u rp ose of this flag is to allow searching and rep orting based on that cap ability. It is a good general p ractice to d isable this p olling if there is no u se for this extra d ata in the object d atabase. V7.51 Interm ed iate Patch 18 introd u ced the -k configPollDelay <#> op tion to allow a d elay of <#> second s for the configu ration p oll if the state of the SN MP ad d ress is d ow n w hen the p oll initiates. This w as introd u ced to ad d ress the case w here a system w ith m u ltip le SN MP su bagent took som e tim e for all su bagents to start after a reboot in ord er to p op u late the interface tables p rop erly. SN MP queries issued by net mon Qu eries issu ed d u ring a netmon d iscovery/ cap ability p oll. This list m ay vary w ith N N M version: .1.3 .1.3.6.1.2.1.1.2.0 .1.3.6.1.2.1.1.1.0 .1.3.6.1.2.1.1.5.0 .1.3.6.1.4.1.11.2.3.1.6.1.1.2.1.2 .1.3.6.1.2.1.17.1.1.0 .1.3.6.1.2.1.17.1.2.0 .1.3.6.1.2.1.17.1.3.0 .1.3.6.1.2.1.4.20.1.1 .1.3.6.1.2.1.4.20.1.3 .1.3.6.1.2.1.4.20.1.2 .1.3.6.1.2.1.2.1 .1.3.6.1.2.1.2.2.1.1 .1.3.6.1.2.1.2.2.1.3 .1.3.6.1.2.1.2.2.1.2 .1.3.6.1.2.1.2.2.1.8 .1.3.6.1.2.1.2.2.1.7 .1.3.6.1.2.1.31.1.1.1.1 .1.3.6.1.2.1.31.1.1.1.18 Test for SN MP Support sysObjectID sysDescr sysN am e hpClusterTable d ot1d BaseBrid geAd d ress d ot1d BaseN um Ports d ot1d BaseType getnext ipAd EntAd d r ipAd EntN etMask ipAd EntIfInd ex ifN um ber ifPhysAd d ress ifType ifDescr ifOperStatus ifAd m inStatus ifN am e ifAlias 49 Fognet’s Field Guide to OpenView NNM .1.3.6.1.2.1.17.1.4.1.1.0 .1.3.6.1.2.1.17.1.4.1.2.0 .1.3.6.1.2.1.4.1.0 .1.3.6.1.2.1.1.6.0 .1.3.6.1.2.1.1.4 .1.3.6.1.2.1.4.21.1.7.0.0.0.0 .1.3.6.1.4.1.11.2.17.4.1.1.1.0 .1.3.6.1.4.1.11.2.36.1.1.1.1.0 .1.3.6.1.2.1.4.21.1.7 .1.3.6.1.2.1.4.21.1.8 .1.3.6.1.2.1.4.21.1.2 .1.3.6.1.2.1.4.21.1.1 .1.3.6.1.2.1.4.21.1.11 .1.3.6.1.2.1.4.22.1.3 .1.3.6.1.2.1.4.22.1.2 .1.3.6.1.2.1.4.22.1.1 .1.3.6.1.2.1.4.22.1.4 .1.3.6.1.2.1.3.1.1.3 .1.3.6.1.2.1.3.1.1.2 .1.3.6.1.2.1.3.1.1.1 .1.3.6.1.2.1.16 .1.3.6.1.2.1.16.20 .1.3.6.1.2.1.10.18 .1.3.6.1.2.1.10.30 .1.3.6.1.2.1.10.32 .1.3.6.1.2.1.10.39 .1.3.6.1.2.1.37 .1.3.6.1.4.1.9.9.23.1.3.1 .1.3.6.1.2.1.10.166.2.1.9 .1.3.6.1.3.96 .1.3.6.1.4.1.9.9.106.1.2.1.1.11 .1.3.6.1.2.1.55 .1.3.6.1.2.1.14.1.1 .1.3.6.1.2.1.15.3.1.1 .1.3.6.1.2.1.68 .1.3.6.1.2.1.17.2.1 .1.2.840.10036 d ot1d BasePort d ot1d BasePortIfInd ex. ipForw ard ing sysLocation sysContact ipRouteN extHop hpOVDStnType m anagem entURL ipRouteN extHop ipRouteType ipRouteIfInd ex ipRouteDest ipRouteMask ipN etToMed iaN etAdd ress ipN etToMed iaPhysAd d ress ipN etToMed iaIfInd ex ipN etToMed iaType atnetAd d ress atPhysAdd ress atIfInd ex rm on m ib rm onConform ance DS1 m ib DS3 m ib fram eRelay m ib SON ET m ib ATM m ib cd pGlobalRun m pls Cisco m pls H SRP IPV6 OSPF BGP4 vrrpMIB STP w ireless Qu eries issu ed d u ring a netmon top ology p oll: ifMib (RFC 1573): .1.3.6.1.2.1.31.1.1.1.1 ifXTableifN am e Mau MIB (RFC 1515): .1.3.6.1.2.1.26.1.1.1.1 .1.3.6.1.2.1.26.1.1.1.2 .1.3.6.1.2.1.26.1.1.1.4 .1.3.6.1.2.1.26.4.x rpMauTablerpMauGroupInd ex rpMauPortInd ex rpMauType d ot3MauType Rep eater MIB (RFC 1516): .1.3.6.1.2.1.22.3.3.1.1.1 .1.3.6.1.2.1.22.3.3.1.1.2 .1.3.6.1.2.1.22.3.3.1.1.5 .1.3.6.1.2.1.22.3.3.1.1.4 50 rptrAd d rTrackGroupInd ex rptrAd d rTrackPortInd ex rptrAd d rTrackN ew LastSrcAd d r rptrAd d rTrackSourceAd d rChanges Discovery .1.3.6.1.2.1.22.3.3.1.1.6 rptrAd d rTrackCapacity Rep eater MIB (RFC 2108): .1.3.6.1.2.1.22.1.4.1.1.1 .1.3.6.1.2.1.22.1.4.1.1.2 .1.3.6.1.2.1.22.1.4.1.1.3 .1.3.6.1.2.1.22.3.1.1.1.1 .1.3.6.1.2.1.22.3.1.1.1.2 .1.3.6.1.2.1.22.3.1.1.1.3 .1.3.6.1.2.1.22.3.1.1.1.4 .1.3.6.1.2.1.22.3.1.1.1.5 .1.3.6.1.2.1.22.3.1.1.1.6 .1.3.6.1.2.1.22.3.1.1.1.7 .1.3.6.1.2.1.22.3.3.1.1.1 .1.3.6.1.2.1.22.3.3.1.1.2 .1.3.6.1.2.1.22.3.3.1.1.5 .1.3.6.1.2.1.22.3.3.1.1.4 .1.3.6.1.2.1.22.3.3.1.1.6 rptrInfoTablerptrInfoId rptrInfoRptrType rptrInfoOperStatus rptrAd d rSearchTablerptrAd d rSearchLock rptrAd d rSearchStatus rptrAd d rSearchAdd ress rptrAd d rSearchState rptrAd d rSearchGroup rtprAd d rSearchPort rptrAd d rSearchOw ner rptrAd d rTrackGroupInd ex rptrAd d rTrackPortInd ex rptrAd d rTrackN ew LastSrcAd d r rptrAd d rTrackSourceAd d rChanges rptrAd d rTrackCapacity Brid ge MIB (RFC 1493): .1.3.6.1.2.1.17.1.1 .1.3.6.1.2.1.17.1.2 .1.3.6.1.2.1.17.1.3 .1.3.6.1.2.1.17.1.4.1.1 .1.3.6.1.2.1.17.1.4.1.2 .1.3.6.1.2.1.17.2.15.1.1 .1.3.6.1.2.1.17.2.15.1.3 .1.3.6.1.2.1.17.2.15.1.8 .1.3.6.1.2.1.17.2.15.1.9 .1.3.6.1.2.1.17.4.3.1.1 .1.3.6.1.2.1.17.4.3.1.2 .1.3.6.1.2.1.17.4.3.1.3 d ot1d Basedot1d BaseBrid geAd d ress d ot1d BaseBrid geN um Ports d ot1d BaseBrid geType.1 d ot1d BasePort d ot1d BasePortIfInd ex d ot1d StpPort d ot1d StpPortState d ot1d StpPortDesignated Brid ge d ot1d StpPortDesignated Port. d ot1d TpFd bTabledot1d TpFd bAdd ress d ot1d TpFd bPort d ot1d TpFd bStatus 51 6. Status Polling - netmon This section covers the fu nctions w ithin netmon that d eliver statu s. netmon is also resp onsible for d iscvoery and configu ration p olling, and those fu nctions are d iscu ssed in Section 5, Discovery. ICMP and SN MP idiosyncrasies and “brow n-outs” When a nod e is not resp ond ing to p ings or SN MP requ ests, N N M generates events and sets top ology statu s. Bu t there are other reasons that m ay cau se ICMP or SN MP requ ests to fail that m ight p rod u ce the sam e events and statu s in N N M w hen in fact the nod e m ay be p erfectly healthy. Som e N N M u sers m ight change the text of statu s events from “nod e d ow n” to “nod e not resp ond ing to p olls” to m ore accu rately reflect this reality. The list below p rovid es ju st som e exam p les. Routers d rops packets d ue to high CPU utilization coupled w ith low priority for ICMP and / or SN MP. Assign higher priorities to ICMP and / or SN MP on routers. Check out Cisco‟s Priority Queueing. N etwork congestion over WAN links is slow ing response tim es. For exam ple, if polls are traversing fram e relay links, then check FECN , BECN , and packets m arked DE. If an N N M status poll goes through a T1 link running AMI, the packet is d ropped. This is d ue to the fact that NN M‟s ICMP payload s are set to 64 bits of all zeros, and the old er AMI cod ing, which is intend ed for use in voice cir cuits, is d esigned to m ake sure there is at least one 1-bit for every 8 bits to red uce potential line harm onics. The solution is to set up the links affected to use B8ZS instead of AMI cod ing. The route that SN MP or ICMP packets take to the source and from it back to the N N M server can be d ifferent, and at least one router m ay be configured to d rop the protocol som ew here along the w ay. Use traceroute or tracert, Problem Diagnosis, rping, and rnetstat to d eterm ine the source of the problem . Adjusting ICMP and SN MP status polling intervals Both SN MP and ICMP p olling intervals are set in the SN MP Configu ration GUI for netmon-based p olling settings. ET-based p olling intervals and other settings are configu red throu gh XML files, the 52 Status Polling - netmon p rim ary of w hich is p aConfig.xm l. Like netmon, ET p olling configs are com bined for both statu s and top ology/ d iscovery. SN MP configu ration settings are em p loyed exactly as configu red . ICMP configu ration settings, how ever, are rou nd ed ou t the nearest second , so the d efau lt tim eou t of 0.8 is actu ally rou nd ed ou t to 1 second for ICMP p olls. The tim eou t interval is re-d ou bled for every retry. A configu ration of 0.8 tim eou t and 2 retries m eans that w hen a n ICMP p oll is issu ed , it shou ld take 7 second s to fail (first p oll tim es ou t in 1 second , first retry tim es ou t in 2 second s, and second retry tim es ou t in 4 second s). An SN MP p oll to the sam e nod e shou ld tim e ou t in 5.6 second s. netmon.noD iscover and netmon.MACnoD iscover files Use these files to enter IP ad d ress and MAC ad d resses ind ivid u ally, by ranges, or by w ild card s to p revent su bsequ ent d iscovery after restarting netmon. Any ad d resses that have been p reviou sly d iscovered w ill have to be m anu ally d eleted from the top ology. To m ake su re an ad d ress is not in the top ology, u se the Find ap p lication from the ovw m enu bar and Find by Attribu te: IP Ad d ress. Entries fou nd can be highlighted from the Find ap p lication and then d eleted from the Ed it m enu bar after choosing View : Select H ighlighted . If p op u lating the netm on.MACnoDiscover file, it m ay be of interest to also set the –k FilterLLAOnlyN od es=tru e LRF flag to the netmon d aem on. Setting this flag w ill force netmon to stop u sing MAC ad d resses in the netm on.MACnoDiscover file as hints for d iscovery. See p age 5 for how to set LRF flags. netmon.interfaceN oD iscover file The file $OV_CON F/ netm on.interfaceN oDiscover is u sed to d efine how to restrict the set of interfaces to be d iscovered for a nod e. This configu ration file w as introd u ced in interm ed iate p atch 12 to V7.5. In V7.51, the InterfaceFiltering.p d f w hite p ap er w a s ad d ed in $OV_DOC.WhitePap ers. In V7.53, this inform ation w as rolled into the Using_Extend ed _Top ology.p d f u ser m anu al. Unlike netm on.noDiscover, netm on.interfaceN od iscover w ill tru ncate the SN MP tables u sed to p op u late interface d ata on connector nod es, greatly im p roving d iscovery and p ollin g p erform ance. N ot only can sp ecific interfaces be exp licitly exclu d ed (exam p le #2 below ), bu t in 53 Fognet’s Field Guide to OpenView NNM tru ncate m od e (exam p le #1 below ), no ad d itional interfaces w ill be su bsequ ently d iscovered once a sp ecified interface is d iscovered . Be carefu l w ith exam p le #1 below (tru ncate) if you are not qu ite certain that all u ninteresting interfaces w ill have a high lighted ifInd ex than the first one m atching the exam p le. Filters can be d efined that are exclu sive or inclu sive. Im p ortant: netm on.interfaceN oDiscover is u sed both by netmon and ET/ APA, so interfaces d efined in netm on.interfaceN oDiscover w ill be d eleted both from the the nod e object in N N M and in ET. If rem oving a large nu m ber of interfaces (thou sand s), consid er not tak ing too m any at the sam e tim e becau se the p rocess of d eleting interfaces is very CPU-intensive. Consid er stop p ing netm on tem p orarily, let ET finish d eleting the interfaces and then start u p netm on again. In the file, an asterick rep resents any nu m ber of characters u p to the next p eriod . For exam p le, *.corp .com m atches p c.corp .com . A qu estion m ark m atches a single character. Brackets m atch a single character, characters in a range, or characters not w ithin a range if '!' is the first character w ithin the brackets. For exam p le: [bf]an.fognet.com m atches ban.fognet.com and fan.fognet.com ; [b-d ]an.fognet.com m atches ban.fognet.com , can.fognet.com and d an.fognet.com ; and [!cz]an.fognet.com m atches only aan.fognet.com and ban.fognet.com . Use the netmon –a 117 com m and to trace netmon’s p arsing of the netm on.nod iscover and netm on.noInterfaceDiscover files. See p age 67 for d etails on netmon tracing. Below are som e exam p les. There are m ore exam p les in the w hite p ap er m entioned above: # Truncate the interface table when an interface with # ifType of 135 is found on node 10.162.191.146 10.162.191.146 1 ifType=135 # Ignore the interfaces with ifType of 135 or 53 for # nodes within the address range: 10.162.191.* 2 ifType=135,53 # For nodes with a name ending in "core", # Ignoreinterfaces with ifDescr containing the string # "VLAN" or "virtual" for node names matching ―core: *core 2 ifDescr="*VLAN*","*virtual*" 54 Status Polling - netmon Commonly used net mon LRF sw itch settings There are d ozen s of LRF sw itches for netmon, each a layered bit of ad d ed fu nctionality that has been ad d ed w ith each su bsequ ent version of N N M. Som e sw itches are p reserved for backw ard com p atibility and are obsolete. Som e of those p ertain to Layer 2 top ology rep resentation. See p age 5 for d etails on u sing the LRF u p d ate p roced u re. -b <num> -q <num> -Q <num> -H -k -k -k Send burst of <num> pings, 1 per sec after retries Increase ICMP receive queue length, see man/ ref Increase SN MP receive queue length, see man/ ref N ote: –q or –Q set too high m ay overload buffers 0 Disable H TTP polls to speed configuration checks bridgeMIB=false Red uce segm ents created from RFC 1493 snmpTimeoutImplies=unknown Default is critical nonIPStatusPolls=false Red uces SN MP sw itch queries Be su re to carefu lly read the m an or reference p age entries for these and other netmon LRF sw itches since changing them can p rofou nd ly change the IPMAP top ology and the granu larity of st atu s p olls. net mon global status polling default The global d efau lt for d iscovery p olling d efined in SN MP configu ration is 15 m inu tes, bu t the object-based p olling introd u ced in N N M version 6.2 overrid es this for m ost d evices of interest (rou ters, sw itches, etc). net mon object-based polling (V6.2+) Object-based p olling allow s d ifferent intervals for d ifferent d evices classes. It also allow s d ifferent intervals for p rim ary vs. second ary interfaces as d eterm ined by netmon‟s critical p ath analysis (m ore on this below ). Objects are d efined via N N M‟s stand ard filter d efinition langu age, and accessed via the Poll Objects configu ration GUI, available as a bu tton in the p olling configu ration. This GUI is a frontend to the netm on.statu sIntervals file. In general, object-based p olling tightens the d efau lt p olling intervals for Rou ters, Brid ges, H u bs, and loosens the d efau lt p olling intervals for N od es to 1 hou r (V6.4+). D ynamically-adjusting status polling by net mon Prior to N N M V6.0, netmon statu s p olling w as static. N N M V6.0 introd u ced the first d ynam ic p olling enhancem ent to su p p ort 55 Fognet’s Field Guide to OpenView NNM ConnectorDow n ECS and the critical p ath analysis, w here p olls to nod es flagged as second ary failu res w ou ld be red u ced by the statu s red u ction m u ltip lier, w hich equ als 2 by d efau lt. This m eans the p olling interval to these nod es w ill be m u ltip lied by tw o as long as the p rim ary nod e rem ains d ow n. V6.31 introd u ced tw o im p ortant netmon enhancem ents that d ynam ically ad ju st p olling intervals to su p p ort new event correlations. These are bu ilt into netmon, so they shou ld continu e to take affect even if those new correlations are d isabled . This first new d ynam ic p olling change in V6.31 is if one int erface changes on a connector, then all of that connector‟s interfaces are im m ed iately p olled . This is non-configu rable and affects any nod e in the object d atabase w ith the cap ability flag isConnector set. This second new d ynam ic p olling change in V6.31 is that any interface on a connector that has changed statu s is re-p olled accord ing to the tw o new netmon –k sw itches below (configu red via LRF, see p age 5): -k shortPollTime=120 -k shortPollDownCount=2 The shortPollTim e is the interval for the re-p oll, and the shortPollDow nCou nt is the nu m ber of re-p olls to issu e. Per the d efau lts show n above, all connector interfaces changing statu s w ill be re-p olled at 2 m inu tes and again at 4 m inu tes after the statu s change. net mon layer 2 polling algorithms Statu s based on layer 2 is very lim ited in netmon w hen com p ared to Extend ed Top ology. netmon u tilizes SN MP qu eries to Brid ge, MAU, Rep eater and som e VLAN MIBs to d eterm ine layer 2 statu s. Un-nu m bered interfaces are inferred from the p ort table, then p olled via ARP. Contrast this w ith ET w hich has p rop rietary MIB extension d ata for hu nd red s of d evices. For netmon, in V5-V6.1, d efau lt statu s for “d ow n” p orts is Critical. In N N M V6.2 throu gh V6.41, the d efau lt statu s for “d ow n” p orts is Unknow n. In N N M V7.0 and above, layer 2 p olling is d isabled in netmon and layer 2 top ology is conveyed via ET. Becau se ET is not enabled by d efau lt in N N M 7.0 and above, there is no layer 2 p olling of any sort tu rned on by d efau lt in these later 56 Status Polling - netmon versions. Previou s versions, how ever, have layer 2 p olling tu rned on by d efau lt u nd er netmon. netmon layer 2 statu s is d eterm ined by p olling stand ard MIB2 SN MP tables for ifAd m inStatu s and ifOp erStatu s. Defau lt SN MP statu s m ap p ing is based on the table below and is cu stom izable via the netm on.statu sMap p ing in N N M 6.2+. In versions p rior to V6.2, statu s m ap p ing is fixed p er the table: ifAd m inStatus ifOperStatus OV Status ---------------------------------------------------------------------dow n any DISABLED testing any TESTIN G up up NORMAL up dow n CRITICAL up testing TESTIN G APA-based layer tw o statu s is entirely d ifferent. All alarm s are log only by d efau lt and top ology statu s is only reflected u nd er Dynam ic View s m ap s. The only statu ses reflected are: Up (green), Contents Dow n (yellow ), Unknow n (second ary failu re), and Disabled (ad m inStatu s is d ow n) (brow n). N on-IP unconnected port status (net mon) The d efau lt statu s for non -IP (layer2) u nconnected interfaces (u nconnected sw itch p orts for exam p le) is Unknow n instead of Critical since N N M 6.2, regard less of ifOp erStatu s and ifAd m inStatu s and netm on.statu sMap p ing file settings. To restore the p re-V6.2 behavior set either or both of the follow ing LRF sw itches in netmon (p age 5): -k ConnectorL2Ports=legacy -k nonConnectorL2Ports=legacy In N N M 7.53 Interm ed iate p atch 19, a new op tion for netm on's -k connectorL2Ports p aram eter of "connected " w as created w hich cau ses netmon to check for connectivity w ithin the top ology d atabase. By d efau lt, netm on u ses brid ge forw ard ing table to d eterm ine connectivity. net mon SN MP-based status for nodes The ability to p oll for p rim ary statu s via SN MP vs. ICMP w as introd u ced to allow p olling across firew alls w here ICMP is not allow ed bu t SN MP m ay be. It is very lim ited in its abilities and is exclu sive – that is if a d evice is p olled for p rim ary statu s via SN MP, it shou ld no longer be p olled by ICMP. 57 Fognet’s Field Guide to OpenView NNM netm on.snm p Statu s is the configu ration file for d efining Level 3 IP ad d ress ranges to p oll via SN MP. Use the follow ing netmon –k LRF sw itch (see p age 5) to control the behavior of SN MP p olling tim eou ts: –k snmpTimeoutImplies=[status] w here statu s is one of: u nknow n, u nchanged , critical (d efau lt). D etermine net mon-based polling intervals for a device The follow ing com m and show s the base p olling settings for a given nod e. It d oes not rep ort on any d ynam ic p olling changes that m ight be in effect: xnmsnmpconf –resolve <node> net mon critical path analysis netmon m aintains an in-m em ory rou te to every interface. This list is only u p d ated w hen the netmon d aem on is started , bu t a d em and p oll (nmdemandpoll) forces a critical p ath recalcu lation for a given p ath if issu ed to any nod e on that p ath. When an interface d oesn't resp ond , netmon looks at all interfaces w ithin th e critical rou te and m arks the closest as p rim ary and all the d istance ones as second ary failu res. ECS correlates su bsequ ent alarm s to the p rim ary failu re u sing the ConnectorDow n correlation and netmon increases the statu s p olling interval for second ary failu res. Failu re statu s for second ary nod es is configu rable throu gh the p olling configu ration. The Im p ortant N od e filter d efines nod es that are alw ays to be consid ered p rim ary by netmon. The p ath chosen by netmon to a p articu lar d evice m ay not be the d esirable p ath w hen there are m u ltip le p aths available. To hard cod e a p ath, u se the follow ing netmon LRF sw itch (see p age 5): -c <critical-route-seedfile> To d u m p the com p lete list of all critical rou te p aths m aintained by netmon: snmpwalk <server> .1.3.6.1.4.1.11.2.17.4.4.2.1.1.5 To d u m p verbose inform ation abou t the target $OV_LOG/ netm on.trace, inclu d ing the critical p ath: netmon –n <target> 58 nod e to Status Polling - netmon To d u m p verbose inform ation abou t the target $OV_LOG/ netm on.trace, inclu d ing the critical p ath: nod e to netmon –i <IP> N ode status events – interpretation IPMAP top ology, w hich is reflected in the ovw top ology m ap s, is hierarchical in natu re. All statu s for IPMAP top ology originates at the interface level, that is, interface-level events are at the root of all other statu s events reflected in the top ology. A “nod e d ow n” event really m eans that all the interfaces w ithin that nod e are either u nreachable or u nkn ow n, etc. The statu s colors reflected on container objects in the top ology is tied to the below list of statu s events u sing the d efau lt statu s p rop agation ru les as set in the m ap p rop erties. Som e of these are logged by d efau lt, m ost of them are not, bu t th e statu s is reflected in the top ology regard less of the logging behavior of the events. OV_N od e_Up: All Ifs are up or unknow n OV_N od e_Warning: One If is d ow n OV_N od e_Marginal: One If is d ow n and >1 If is up OV_N od e_Major: One If is up d ow n OV_N od e_Dow n: All Ifs are down or unknow n OV_N od e_Unknow n: All Ifs are unknow n Handling multiple OV_N ode_Up events The OV_N od e_ Up event is generated by netmon w hen it d etects that all the nod e interfaces are u p . As a sid e-effect, w hen one of the nod e's interfaces goes critical and after a w hile retu rns to norm al, the OV_N od e_Up event is generated w ithou t an int erm ed iary OV_N od e_Dow n. This m ay not be the d esired behavior; an OV_N od e_u p is som etim es only w anted after an OV_N od e_d ow n. (All interfaces d ow n). To change this behavior to there is a one-to-one correspondence between the OV_Node_Up and OV_Node_Down events, ow these steps: Modify the OV_Node_Down event: (Specific 58916865) to add an action callback to the Create_NodeDownFile.ovpl script (listed below). Send varbind $3 and $2 to the script as respective arguments, for example: 59 Fognet’s Field Guide to OpenView NNM OVHIDESHELL Create_NodeDownFile.ovpl $3 $2 Modify the OV_Node_UP event: (Specific 58916864) to add an action callback to the Check_NodeDownFile.ovpl script (listed below). Send varbind $3 and $2 to the script as respective arguments, for example: OVHIDESHELL Check_NodeDownFile.ovpl $3 $2 Create the scripts listed below and assure they are configured as Trusted Commands (See page 157 for details on Trusted Commands). #!/opt/OV/bin/Perl/bin/perl # Create_NodeDownFile.ovpl objectID [information] # Create a file: $OV_TMP/NodeDown_objectID.txt with optional additional information # as contents and also appends info to the $OV_TMP/NodeDown.log file. # (This information can be used to store the IP-address, downtime etc...) use OVvars; sub log { my ($logstring)= @_; open (LOGFILE, ">>$OV_TMP/NodeDown.log") || die ("Cannot create/open $OV_TMP/NodeDown.log\n"); print LOGFILE "$logstring\n"; close LOGFILE; } if ($#ARGV >= 0) { $objectID = $ARGV[0]; $downfile = "$OV_TMP/NodeDown_$objectID.txt"; } else { &log ("Create_NodeDownFile.ovpl called without parameters"); exit 1; } open (DOWNFILE, ">$downfile") || die ("Cannot create/open $downfile\n"); if ($#ARGV >= 1) { $information = $ARGV[1]; print DOWNFILE ($information); 60 Status Polling - netmon } &log ("Created $downfile $information"); close DOWNFILE; exit 0; #!/opt/OV/bin/Perl/bin/perl # Check_NodeDownFile.ovpl object_ID [information] # Check if a file: $OV_TMP/NodeDown_objectID.txt has been created previously. # If present, a real OV_Node_Down has occured previously and now a real OV_Node_Up # message should be displayed in the Alarm browser and the file has to be deleted to rearm # the mechanism. The file contents and given information can be used as desired. # (e.g. display nodename, calculate the downtime etc.) # Also append info to the $OV_TMP/NodeDown.log file. use OVvars; sub log { my ($logstring)= @_; open (LOGFILE, ">>$OV_TMP/NodeDown.log") || die ("Cannot create/open $OV_TMP/NodeDown.log"); print LOGFILE "$logstring\n"; close LOGFILE; } if ($#ARGV >= 0) { $objectID = $ARGV[0]; $downfile = "$OV_TMP/NodeDown_$objectID.txt"; } else { &log ("Check_NodeDownFile.ovpl called without parameters, can't check filename"); exit 1; } open (DOWNFILE, "$downfile") || die ("Cannot open and check $downfile\n"); $contents = <DOWNFILE>; if ($#ARGV >= 1) { $information = $ARGV[1]; $message = $information; 61 Fognet’s Field Guide to OpenView NNM } else { $message = $contents; } $category = "Status Alarms"; $localhost = "Localhost"; $message = "Node UP event for Object ID $objectID $message"; &log ("$message for category: $category "); $cmd = "ovevent -c \"$category\" \"\""; $cmd .= " .1.3.6.1.4.1.11.2.17.1.0.58916872"; $cmd .= " .1.3.6.1.4.1.11.2.17.2.1.0 Integer 14"; $cmd .= " .1.3.6.1.4.1.11.2.17.2.2.0 OctetString \"$localhost\""; $cmd .= " .1.3.6.1.4.1.11.2.17.2.4.0 OctetString \"$message\""; `$cmd`; close DOWNFILE; unlink "$downfile"; exit 0; Status event variable bindings of interest The table below lists som e netmon interface and nod e event varbind s: IF Status N od e Status Varbind # Varbind # Description -----------------------------------------------------------------------------------------$2 $2 H ostnam e of nod e that caused event $5 $5 Tim estam p event occured $7 Interface N am e or Label $8 IP Ad d ress of Interface or “0” $11 Num ber of bits in the subnet m ask * $12 Interface ifAlias * $13 * $8 Local list of capabilities * $14 * $9 Nam e of prim ary failure host * $15 * $10 Nam e of prim ary failure entity * $16 * $11 OV OID of prim ary failure entity * $17 * $12 Description of prim ary failure entity * $18 * $13 Prim ary failure list of capabilities * = N ew as of N N M V6.31 62 Status Polling - netmon For exam p le, here are tw o statu s event texts from N N M V6.31+: IF $7 Down $12, Capabilities: $13 Root Cause $14 $15 Node Down Capabilities: $8 Root Cause: $9 $10 Contrast these w ith the sam e statu s event texts in version s p rior to V6.31: IF $7 Down Node Down ICMP burst polls The –b <seconds> LRF op tion to netmon (see p age 5 for LRF instru ctions) send s a bu rst of p ings, one p er second to any object after the regu larly-issu ed set of tim eou ts and retries have been exhau sted . ICMP redirects If netmon –a 3 show s m essages sim ilar to this: unexpected ICMP message 5 code 1 from 192.168.1.1 …w here the IP is the N N M Server‟s d efau lt gatew ay, the n the d efau lt gatew ay is p robably im p rop erly configu red . This sort of error cau ses N N M‟s p olling w aitlist to get severely backed u p . More inform ation on ICMP red irects can be fou nd in RFC 792. ICMP typ e 5 m essage cod es are: 0 1 2 3 Redirect Redirect Redirect Redirect datagrams datagrams datagrams datagrams for for for for the Network the Host Type of Service and Network Type of Service and Host ICMP re-d irects m ay also hap p en w hen nod es that are red irected are on the other sid e of an internal rou ter have a rou te to a su bnet that the hosts and are on via a non-p ersistent WAN link. When this WAN link goes d ow n, that rou te d isap p ears from the rou ter's rou te table. So, w hen netmon p olls the nod e, the rou ter send s an ICMP red irect to N N M p ointing it tow ard s another server becau se the rou ter d oesn't have the rou te to th at nod e anym ore. The other server to w hich is red irecting is likely to be the red irecting rou ter‟s ow n d efau lt rou te, w hich is d ifferent than the N N M server‟s d efau lt rou te. Within N N M, the nod e m ay be rep orted d ow n becau se netmon is trying to reach it throu gh the red irecting gatew ay‟s d efau lt rou te, 63 Fognet’s Field Guide to OpenView NNM w hich m ay be a firew all. If this is the case, one solu tion is to have a sep arate segm ent for the firew all from the rou ter w ith no other nod es on it. That w ay, traffic shou ld be forced to go throu gh the internal rou ter regard less of w hether the WAN link to the nod es is d ow n or not. net mon vs. APA poller netmon p rovid es d iscovery, configu ration, and top ology as w ell as statu s p olling. In V7.0 and later, a new APA p oller w as m ad e available to take over netmon’s statu s p olling fu nctions, to take ad vantage of ET‟s m ore com p rehensive p ictu re of statu s via SN MP and to get arou nd netmon‟s lim itations d u e to legacy architectu re. In general, netmon-based p olling is ICMP-based excep t for non -IP and d evices sp ecifically set to be p olled via SN MP in the netm on.snm p Statu s file. APA p olling generally com bines ICMP w ith SN MP-based statu s p olls, is m assively m u lti-thread ed , and the d evice typ e d ictates how the d evice shou ld be p olled . Also, the APA is “neighbor aw are” in how it hand les failu res. The APA how ever, requ ires the extend ed top ology is enabled and is d ep end ent on ET d iscovery to w ork p rop erly. In som e environm ents, ET rem ains p roblem atic to d ep loy and m aintain. netmon Cons Single-thread ed , single protocol, legacy issues (IP, DN S) Polls via ICMP or SN MP, never both, SN MP polls lim ited Second ary status d eterm ination mechanism s rud im entary Rud im entarily d ynam ic in adjusting to neigh bor status Cannot poll into OAD‟s; not good in hand ling H SRP, N AT, etc. Cannot separate IP ad d ress from a physical interface Is not “connection aw are,” rud im entary path analysis Unique path to each interface to determ ine prim ary Disparate configuration sw itches, files, and d epend encies Provid es status at only tw o entity levels: interface, nod e netmon Pros Behavior is 100% characterizable and configurable, less FUD Still scales w ell w hen properly tuned / controlled GUIs available for polling custom ization / configuration Preservation of investm ent in netmon-based status Doesn‟t d epend on ET d iscovery APA Cons 64 Analysis engine com plex, d ifficult to characterize/ control APA requires ET enabing and timely ET d iscovery Custom ization via XML files; no configuration GUI includ ed Status Polling - netmon netmon-based custom izations not inherited by APA, e.g.: - Poller settings in SN MP Configuration GUI ignored - Interfaces unmanaged in N N M topology or by ovautoifm gr still m ay be polled - Object based polling settings based on NN M filters (APA uses ET filters) Separate d efinition files/ filters for Im portant N od es nm d em and poll, ovet_toposet, checkPollCfg N/ A in 7.01 Cannot be used in DIM environment MS (CS only) Incom patible w ith LAN/ WAN Edge & MPLS SPI in 7.01 (ok in 7.5+) IPX polling not available after sw itching to APA Initial setup can cause flood s of events (m ostly log-only) IPMAP topology status from APA can be problem atic Interface count increases if form erly not m anaging Level 2 APA Pros Multi-thread ed , m ulti-protocol (ICMP & SN MP, others) Sw itched -topology-aware, d up IP-aware, neighbor-aware Provid es status at m ultiple entity levels: - Ad d ress, Interface, N od e, Connection , Board, AgPort Provid es m ore d ynam ic polling based on queued status: Provid es connection -oriented and d evice-oriented status Provid es m ore accurate & tim ely status than netmon Less reliant on ECS, more correlation at the source Generates fewer log-only and em bed d ed status events Polls OAD, H SRP, and other IP add ress sharing interfaces Few er, more intelligent and m ore tim ely status events More granularity in entity-based polling via XML files Red uces need to use d istributed NN M Vend or-spesific agents gives better inform ation for various types of equipm ent net mon polling statistics netmon p olling statistics can be view ed from the com m and line by ru nning: ovstatus –v netmon Statistics can also be grap hed from ovw by selecting the N N M server icon in the m ap and ru nning Perform ance > N etw ork Polling Statistics. Trend s of negative nu m bers ind icate that netmon isn‟t keep ing u p w ith its list of objects to be p olled . Very frequ ently, this cond ition on new installations is ind icative of issu es w ith DN S looku p p erform ance. 65 Fognet’s Field Guide to OpenView NNM Figure 6-1 The grap h in figu re 6.1 show s an exam p le trend for a from -scratch d iscovery of a netw ork consisting of 650 interfaces. The top line rep resents statu s p oll list length and u nd er norm al circu m stances this line shou ld rem ain flat. This grap h w as started shortly after a fresh installation and netmon had alread y d iscovered abou t 300 interfaces. The second line from the top , statu s p olls in the next m inu te, abru p tly rises then trend s back to zero. SN MP w as m isconfigu red and the low est line, second s u ntil next SN MP p oll w as w oefu lly behind . After that, the SN MP issu e w as ad d ressed , and netmon w as able to d iscover another 350 interfaces. The second s u ntil next SN MP p oll recovered nicely bu t then the second s u ntil next statu s p oll started to trend negative, a norm al reaction to a bu rst of new -nod e d iscovery. The thick line that hovers below zero is second s u ntil next statu s p oll,. Anytim e that the valu es for this m etric are below zero, it ind icates that som e nod es are not getting p olled w ithin their d efined statu s p olling intervals – netmon is falling behind . Und er norm al op erating cond itions, this line stays near zero. 66 Status Polling - netmon net mon troubleshooting, tracing and dumping Ru n: netmon -M <mask> to send an event to the ru nning netmon to tu rn on tracing. Once tracing is enabled , ru n: nmdemandpoll <node name> or p erform som e other action to get netmon to p erform the task to trace. Trace ou tp u t is w ritten $OV_LOG/ netm on.trace. Ru n: netmon -M 0 to tu rn off tracing as netmon.trace grow s w ithou t bou nd s and can easily fill u p a d isk w ith m ore ver bose tracem asks. Tracem asks are ad d itive, so to trace both SN MP requ ests and SN MP rep lies and tim eou ts, u se a tracem ask of 12 (4+8). Several tracem asks are listed in the netmon m an/ ref p age; the follow ing are som e ad d itional tracem asks: 0x00000100 0xffffffff Trace effects of netmon.cm str file Turn on all m asks – uses lots of d isk/ CPU netmon –a <action-number> d u m p s netmon internal d ata stru ctu res su ch as p ing lists and SN MP lists to $OV_LOG/ netm on.trace. Use the follow ing com m and to d u m p a list of all action-nu m bers and their m eanings: netmon –a ? netmon –n <target> d u m p s verbose inform ation abou t the target nod e to $OV_LOG/ netm on.trace, inclu d ing the critical p ath. netmon –i <IP> d u m p s verbose inform ation abou t the target Interface to $OV_LOG/ netm on.trace inclu d ing the critical p ath. 67 7. Status Polling - APA Introd u ced in N N M 7.01, the APA p olls H SRP and OAD by d efau lt, and can be configu red to take over for netmon statu s p olling. Before enabling APA p olling, read the p ros and cons of netmon vs. APA p olling on p age 64. There w ere several changes to the APA betw een V7.01 and V7.5 and in su bsequ ent p atches to V7.5 and V7.51. These changes concern p erform ance, scalability, granu larity, and the ability to characterize p olling behaviors. If u sing the APA, alw ays m ake su re the latest u p d ates are ap p lied . APA Architecture Polling p olicies are labeled as Class Sp ecifications in the p aConfig.xm l file. The three p rim ary “p aram eters” that are u ser configu rable are snm p Enable, p ingEnable, and interval. For each object (like an ad d ress, interface or nod e), APA begins p arsing the p aConfig.xm l from the top d ow n looking for a p olling p olicy m atch for the object and the p aram eter setting. It also looks for m atch for its p arent objects and ap p lies logic to m ake a d ecision abou t how to p oll the object. For snm p Enable and p ingEnable p aram eters, this is a logical AN D. For interval, its m ore of an if/ else statem ent as d etail below . If it can‟t find an exp licit p olicy m atch for the object, it reverts to the d efau lt setting. N ote that d efau lt settings are “typ eless”, m eaning there isn‟t a sep arate d efau lt for nod es, interfaces and ad d resses. They all share the sam e d efau lt settings. The APA then d oes the sam e algorithm for the p arent object and p erform s a logical AN D of all the resu lts. In APA, N od e is the p arent of Interface w hich is the p arent of Ad d ress. N od e <- Interface <- Ad d ress To find a p olling p olicy for an ad d ress in p aConfig.xm l, it w ill look 68 Status Polling - APA for exp licit p olling p olicies for the ad d ress, the p arent interface and the p arent nod e. It w ou ld look som ething like this: (Ad d ress Setting) AN D (Interface Setting) AN D (N od e Setting) So, for a given ad d ress, APA begins looking throu gh the p aConfig.xm l file for a p olicy m atch for the ad d ress. If it d oesn‟t find one, it reverts to the d efau lt setting w hich is PingEnable=tru e. N ow it goes to the p arent object w hich is an interface. Su p p ose it m atches on “u nconnected interfaces on rou ters” and that configu ration is set to PingEnable=false. N ow it goes to the p arent object w hich is a nod e and it find s one for rou ters and the configu ration is set to PingEnable=tru e. AN D all these valu es together and you ‟ll get: Tru e AN D False AN D Tru e = False Therefore the ad d ress in not p inged . N ow , for the interval p aram eter, the logic is sim ilar bu t rather than AN Ding the valu es together, it ju st takes the first valu e it m atches and d oesn‟t try the d efau lt u ntil all p arents are exhau sted . So the logic is: (Ad d r Interval) else (Intf Interval) else (N od e Interval) else (d efault) For exam p le, a set of rou ters has a p olling interval of 120 second s and there are no p olling intervals set for any Interfaces and no p olling intervals set for ad d resses. Su p p ose the d efau lt p olling cycle is set to 300 second s: (u nd efined ) else (u nd efined ) else (120s) else (500s) = 120s APA vs net mon status architecture The follow ing d iagram show s the m ajor su bsystem s of both legacy N N M and ET and their relationship s to statu s. This relates only to N N M 7.x versions: 69 Fognet’s Field Guide to OpenView NNM 70 Status Polling - APA APA and Smart Plug-Ins Both the Sm art Plu g-in for Fram e Relay (version 2.0) and the Sm art Plu g-in for MPLS IP VPN (version 1.0) are incom p atible w ith the APA and requ ire netm on to be the d aem on that p olls objects in ord er to w ork p rop erly. Do not enable the APA if these p lu g -ins w ere p u rchased . Turn APA polling on or off (“the big sw itch”) To enable APA, follow these step s: 1. Read $OV_DOC/ w hitepapers/ Active_Problem_Analyzer.pd f or the Using_Extend ed_Topology user m anual after V7.53 2. Exit GUI sessions 3. Run: setupExtTopo.ovpl, then etrestart.ovpl. Wait 4. Run ovet_apaConfig.ovpl -enable APAPolling 5. Watch ovstatus carefully : ovstatus –v netmon Should say “Polling 0 interfaces” ovstatus –v ovet_poll Should say “Polling d evices” 6. If netmon is still polling, run: ovstop and ovstart To d isable the APA, ru n: ovet_apaConfig.ovpl -disable APAPolling D etermine poller control To u nam bigu ou sly d eterm ine w hether netmon is p olling for statu s or the APA, ru n: ovet_apaConfig.ovpl -query APAPolling APA configuration in a nutshell The follow ing xm l files are u sed to configu re the APA: $OV_CON F/ $OV_CON F/ $OV_CON F/ $OV_CON F/ nnm et/ nnm et/ nnm et/ nnm et/ paConfig.xml topology/ filter/ TopoFilters.xm l topology/ filter/ APANoPollN od es.xm l topology/ filter/ MyH ostID.xm l The follow ing tools are u sed to tu ne APA settings: /opt/OV/bin/ovet_topodump.ovpl /opt/OV/support/NM/checkpollcfg /opt/OV/bin/ovet_demandpoll.ovpl 71 Fognet’s Field Guide to OpenView NNM D emand polling using ovet_demandpoll.ovpl The GUI-based “p oll nod e” and the u nd erlying com m and nmdemandpoll shou ld force a netmon d iscovery p oll, bu t not a statu s p oll for APA nod es. In N N M 7.01 and u p , ovet_demandpoll.ovpl is available for forcing APA statu s p olls. If issu ed w ith the –d op tion, d etailed inform ation abou t the internal states of the d evice being p olled w ill be d u m p ed , bu t the actu al d ev ice w ill not be actively p olled . Below are all op tions, som e of w hich are not su p p orted in earlier releases of N N M: -o -d -t -r -V -v -B -s -P -p -g Sp ecify a sp ecific object (su p p ly OID) Du m p the statu s of the object st Tim eou t to w ait for 1 resp onse Sp ecify the Overlap p ing Dom ain ID Verbose Use to show H SRP virtu al IP Force sync betw een legacy and ET d atabases Generate su m m ary rep orts Enable logging of p olling activity (requ ires –o) Disable logging of p olling activity (regu ires –o) Create sam p le island Grou p config file The –g op tion above w as ad d ed in N N M 7.53 and is d iscu ssed in the section below on Island Grou p Monitoring. The –d op tion above changes after installing N N M 7.51 Interm ed iate Patch 15, p rovid ing m ore d etailed info on u se of ET filters. Exam p le ovet_demandpoll.ovpl –d <node> ou tp u t: OBJ_TYPE SHORT_OBJECT_NAME snmpEnable ET_FILTER ------------------------------------------------------------all-types * true DEFAULT NODE nodename true isRouter Composite nodename true Node Value OBJ_TYPE SHORT_OBJECT_NAME pingEnable ET_FILTER ------------------------------------------------------------all-types * true DEFAULT NODE nodename true isRouter Composite nodename true Node Value OBJ_TYPE SHORT_OBJECT_NAME interval ET_FILTER ------------------------------------------------------------all-types * 300 DEFAULT NODE nodename 300 isRouter Composite nodename 300 Node Value 72 Status Polling - APA N ote in the above ou tp u t, the resu lts show the SN MP, Ping, and interval settings for each interface on the nod e. Exam p le ovet_demandpoll.ovpl –d <interface> ou tp u t: OBJ_TYPE SHORT_OBJECT_NAME snmpEnable ET_FILTER ------------------------------------------------------------all-types * true DEFAULT NODE nodename true isRouter INTERFACE nodename[0[1]] true UnconnectedIF Composite nodename[0[1]] true InterfaceVal If ovet_demandpoll.ovpl is issu ed w ith the –V op tion, d etailed inform ation w ill be d u m p ed abou t the act ively issu ed p oll. If issu ed w ith the –B op tion, the statu s brid ge w ill be force-u p d ated . Use this to help u p d ate stale statu s that is show ing in the IPMAP top ology or if the Dynam ic View s m ap s show inconsistent statu s w ith the ovw m ap ‟s statu s for a p articu lar object. On w ind ow s, the ovet_d em and p oll.ovp l scrip t calls an execu table called $OV_SUPPORT/ N M/ ovet_d em and p oll.exe. Du e to the w ay this is stru ctu red , ou tp u t from the scrip t can‟t be red irected to stand ard ou tp u t. To w ork arou nd this, cop y ovet_d em and p oll.exe into the $OV_BIN d irectory and then ru n ovet_d em and p oll.exe instead of ovet_d em and p oll.ovp l. The argu m ents are id entical and the ou tp u t can be red ict4ed , for exam p le to a file: ovet_d emand poll.exe –d patchy.fognet.com > c:\ tem p\ patchy.dm p APA Island Group Monitoring Introd u ced in N N M 7.53, this featu re p rovid es a m ethod for hand ling “island s of connectivity” that resu lt from ET‟s inability to d iscover connectivity throu gh WAN clou d s. The featu re is enabled by d efau lt, bu t configu ration of the island grou p nam es is necessary in ord er to p rop erly id entify the island grou p nam es that N N M w ill rep ort on. This rep orting is conveyed throu gh the follow ing tw o new APA events: OV_APA_ISLAND_GROUP_DOWN OV_APA_ISLAND_GROUP_UP The Grou p Dow n alarm m ight look like the follow ing: Rem ote Site N od e Group Dow n 3 Capabilites: isIPRouter,isSw icth, 73 Fognet’s Field Guide to OpenView NNM The 2 varbind s p assed after the “Grou p Dow n” text are the grou p nu m ber and the grou p nam e, w hich by d efau lt m ean little u nless those attribu tes are exp licitly cod ed p er the p roced u re below . ET d iscovers nod es and their connectivity, bu t not the grou p s. The APA bu ild s grou p s w hen load ing d ata from ET. By d efau lt, a single nod e in the grou p , typ ically a rou ter is selected to rep resent the grou p . A label w hich by d efau lt is blank is then available to nam e the grou p . To cu stom ize this behavior, follow these step s: 1. 2. 3. Run ovet_d em and poll.ovpl –g to create a sam ple configuration file called $OVCON F/ nnm et/ paIsland GroupN am eSam ple.txt Ed it the file to ad d group nam es and , if d esired , change the representative nod e for the group. The d efault nam e chosen by APA can be used if d esired . The nam e m ay contain spaces. Restart ovet_poll using ovstop/ovstart WAN connections w here ET can d eterm ine connectivity, like fram e relay links w here CDP is ru nning over them , w ill not be consid ered island s. Bu t rem ote sites that are connected throu gh MPLS clou d s w ill be consid ered island s, since ET can‟t resolve that p articu lar sort of connectivity. Island Grou p Monitoring is enabled (d efau lt) or d isabled via the p aConfig.xm l file p aram eter disableIslandGroupDiscovery. APA status events APA-generated statu s events are d ifferent from netmon-based statu s events, for exam p le: OV_APA_IF_DOWN (Op enView enterp rise sp ecific event nu m ber 58983012). With APA p olling, ICMP and / or SN MP p olls relate to ad d ress, interface, Aggregated Interface, N od e, Connection, and Board . In the APA, p olling granu larity is d efined by ET Top ology filters. “Unreachable” is the term u sed in the APA to d escribe second ary entity failu re statu s. APA status from SN MP traps APA listens for the below SN MP trap s generated by d evices to sp eed and ad d intellignce to the APA statu s. In m any environm ents, these trap s are d isabled at the sou rce to red u ce SN MP chatter. Assu ring that these trap s are tu rned on at the d evice level for critical d evices m ay increase the sp eed and accu racy of APA statu s. CiscoColdStart CiscoLinkDown CiscoLinkUp CiscoWarmStart 74 1.3.6.1.6.3.1.1.5.1 1.3.6.1.6.3.1.1.5.3 1.3.6.1.6.3.1.1.5.4 1.3.6.1.6.3.1.1.5.2 Status Polling - APA ColdStart WarmStart HSRPState LinkDown LinkUp chassisChangeNotifOID StackMIBModuleDown StackMIBModuleUp 1.3.6.1.6.3.1.1.5.1 1.3.6.1.6.3.1.1.5.2 1.3.6.1.4.1.9.9.106.2.0.1 1.3.6.1.6.3.1.1.5.3 1.3.6.1.6.3.1.1.5.4 1.3.6.1.4.1.9.5.11.2.0.2 1.3.6.1.4.1.9.5.0.4 1.3.6.1.4.1.9.5. Suppress/allow APA status polling To su p p ress or allow APA p olling to interfaces/ p orts u sing Dynam ic View s, u se the Port Ad m in Tool (p age 249) if ru nning N N M V7.53. To su p p ress or allow p olling to any APA -p olled entity (nod es, interfaces, connections, board s and ad d resses), u se the ovet_toposet.ovpl com m and . This w ill w ork only for APA nod es in N N M 7.50 and u p : -a -s -o Option allow s APA polling for that entity Option suppresses APA polling for that entity Specify a particular object as the target Use this com m and in conju nction w ith ovtopofix -G to u nm anage d evices on the netmon sid e. N od es that have been su p p ressed w ill eventu ally be rem oved from both the ET and legacy d atabases. In typ ical environm ents, this takes abou t 7 d ays. For exam p le, to stop p olling to a p articu lar interface on a nod e: 1. Confirm polling status of interfaces by running: $OV_SUPPORT/NM/checkPollCfg –o <node> 2. To m ap interface IP to interface nam es, run: ovet_topodump.ovpl –nodeif <node> When the interface nam e is too long, for exam ple “H P NC7781 Gigabit Server Ad apter,” the OID string can be used instead of the ifNam e string w hen specifying the interface in the com m and ovet_toposet below . To get the OID string, run: ovet_topodump.ovpl –nodeif –detail <node> 3. Disable polling on interface hme1: $OV_SUPPORT/NM/ovet_toposet –s –nodeif <node> -if hme1 4. Repeat step 1 to confirm polling configuration change APA status events Selected APA varbind s of interest: Varbind # Description 75 Fognet’s Field Guide to OpenView NNM --------------------------------------------------------------------------$2 Tim estam p event occured $3 H ostnam e of nod e that caused the event $5 Label of the responsible interface $6 ifAlias of the responsible interface $8 ifInd ex of the responsible interface $9 ifDescr of the responsible interface $10 Responsible Level 3 add ress or port # $11 Responsible Level 2 add ress $12 Subnet Mask $13 Route Distinguisher $15 Capabilities $16-$28 Double-object failure varbind s if connector $29-$42 Prim ary failure varbind s if secondary Exam p le APA Event texts: IF Down: Address Down: Connection Down: Node Down: $5 $10 $6 Capabilities: $15 $5 $10 $6 Capabilities: $15 $5 $10 connected to $16 $18 $10 Capabilities: $15 APA status as reflected in ovw maps APA statu s is m u ch m ore accu rate and rep resentative of the p olling entities in Dynam ic View s than it is in brid ged statu s to the legacy top ology. The ovet_bridge d aem on is resp onsible for m ap p ing APA statu s to ovw m ap statu s. The m ap p ing betw een ET and the ovtop m d is controlled by a flat file called hosts.nnm . This file shou ld not be changed by u sers. Resolve any d escrep ancies throu gh ET d iscovery or throu gh ovw top ology d atabase m anip u lation tools su ch as ovtopofix. Enabling APA for the general IP environm ent can resu lt in interfaces m arked "norm al" in ovtopmd/ ovw that shou ld be m arked "critical." When APA attem p ts to m atch interface objects in ovtopmd w ith interface objects in the Extend ed Top ology d atabase, interface objects that d o not d irectly m atch w ill be m arked "norm al" in the ovtop m d d atabase and ovw, and a m essage noting this w ill be logged in $OV_LOG/ ovet_p oll.log. If an interface exists in the ovtopmd/ ovw d atabases in m u ltip le form s, only one of those interfaces w ill be u p d ated . If Layer-2 connectivity d iscovery is enabled in netmon, netmon w ill attem p t to d o connectivity d iscovery on brid geMIB su p p orting d evices, resu lting in extra interfaces w ith the sam e ifN u m ber u nd er som e cond itions. APA w ill only u p d ate one of these interfaces w ith statu s. Ad d itionally, these interfaces are som etim es d eleted and recreated , resu lting in an inconsistency w ith the Extend ed Top ology 76 Status Polling - APA d atabase u ntil the next ET d iscovery. Sim ilarly, som e interfaces are created to connect sw itches or other connectors to segm ents in the ovtop m d d atabase. These interfaces, w hich have no real interface attribu tes, w ill also be m arked norm al. If Mu ltip le IP Ad d resses exist on the sam e interface, ovtopmd m od els this as tw o interfaces w ith d ifferent IP ad d resses bu t the sam e ifN u m ber. APA and Extend ed Top ology w ill treat this as a single interface w ith m u ltip le IP ad d resses. APA w ill u p d ate only one of the interfaces in the ovtopmd/ ovw d atabase w ith correct statu s. In general, ET re-d iscovery w ill resolve m ost d escrep ancies betw een the ET d b‟s view of the top ology and ovtopmd/ ovw’s. APA aggregated port support N ew er APA events in N N M 7.5 su p p ort AgPort by m ap p ing m u ltip le p hysical p orts via an ET tru nk virtu al p ort. The APA p olls p hysical interfaces, bu t not logical interfaces. Su p p ort is for Cisco PAgP in V7.5. V7.51 ad d ed su p p ort for N ortel MLT and SMLT. V7.51 su p p lied a w hite p ap er on the N ortel agents in: $OV_DOC/ WhitePapers/ MLT.pd f In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. The p reviou s tru nk su p p ort via the netm on.equ ivPorts configu ration file is now called “red u nd ant connection su p p ort.” Fine tu ne AgPort via the follow ing ET Top ology filters: isAggregated IF isPartOfAggregatedIF virtual interface physical interface Aggregate p ort scenario: Suppose one physical port goes d ow n on a trunk. In this case, a TrunkDegrad ed event issued and the trunk virtu al port status changes to m inor in ET. Also, the physical interface changes to critical and the Interface Dow n APA event is correlated and em bed d ed by the ConnectorDow n correlation . AgPort status from ET is not propagated to the IPMAP topology and can only be observed in Dynam ic View s. AgPort statu s events: 77 Fognet’s Field Guide to OpenView NNM OV_APA_AGGPORT_DEGRADED: the aggregate port connection betw een tw o nod es is respond ing to polls and som e of the interfaces are d ow n. OV_APA_AGGPORT_DISABLED: the prim ary aggregated port is not respond ing to polls in a norm al fashion. This could be because all the interfaces' ifAd m inStatus are Dow n| Testing. OV_APA_AGGPORT_DOWN : the aggregate port connection betw een tw o nod es is not respond ing to polls and all interfaces on this side of the connection m ay be d ow n. OV_APA_AGGPORT_UN REACH ABLE: the aggregate port conn ection betw een tw o nod es is not respond ing to polls. The problem is d ue to another entity. OV_APA_AGGPORT_CON N _DOWN : the aggregate port conn ection betw een tw o nod es is not respond ing to polls and all interfaces m ay be d ow n on both sid es of the connection. OV_APA_AGGPORT_REMOVED: the SN MP query returned noSuchObj. This can occur if the port is reconfigured of if an ind ex renum bering has occurred . It can also be a problem w ith the system ‟s SN MP agent. This event w as ad d ed in a patch to V7.5. APA HSRP and VRRP status support The N N M Ad vanced Rou ting SPI license is requ ired to u se the H SRP or VRRP ET Device agents that the APA need s to p rovid e p rotocol related statu s. The follow ing statu s events w ere ad d ed in a p atch to V7.5 for VRRP su p p ort: • • • • • • rcVrrpTrapN ew Master rcVrrpTrapAuthFailure rcVrrpTrapStateTransition vrrpTrapN ew Master vrrpTrapAuthFailure snTrapVrrpIfStateChange The follow ing APA H SRP events w ere ad d ed in V7.01 and enhanced for VRRP Protocol su p p ort in a p atch to V7.5: • • • • • • • 78 OV_H SRP_N o_Active OV_H SRP_Multiple_Active OV_H SRP_N oStand by OV_H SRP_Degrad ed OV_H SRP_FailOver OV_H SRP_Stand by_Changed OV_H SRP_N orm al Status Polling - APA • OV_H SRP_Multiple_Stand by N ew event corellations w ere ad d ed in V7.51 to su p p ort VRRP statu s. It is im p ortant to u nd erstand the p oll trigger featu res of these corellators. More on these corellators can be fou nd in: $OV_DOC/ WhitePapers/ VRRP.pd f In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. APA d efau lt configu ration settings affecting H SRP statu s are as follow s: H SRPTransientWait GenerateNoStandbyEvent GenerateDegradedEvent GenerateFailoverEvent GenerateStandbyChangedEvent 60000 true true true true H SRPTransientWait is m illisecond s to w ait for H SRP to becom e stable after a failover. Red u cing this nu m ber increases the chance of u nnecessary statu s events. The rem aining configu ration settings control the generation of interm ed iate statu s events. APA board status support Su p p ort for board statu s is ru d im entary in N N M 7.5. Wid er su p p ort for this featu re can be exp ected in fu tu re versions althou gh no ad d itional su p p ort w as ad d ed in N N M 7.51 or N N M 7.53. As of V7.53, board statu s is only su p p orted for the follow ing MIBs: Cisco Stack MIB Rhino MIB C2900 MIB Su p p ort for OLD-CISCO-CH ASSIS-MIB w as ad d ed in Intem ed iate p atch 15 to N N M V7.51. The Board Dow n event correlation logic, as w ell as APA event triggered p olling based on the Board Dow n trap , w ill not w ork w ith old er versions of the Cisco stack MIB. Any version of the stack m ib that has m od u leTyp e in var bind 1 of the m od u le d ow n trap , instead of m od u leInd ex, w ill not w ork. Confirm the ap p rop riate version of the stack MIB throu gh the event configu ration GUI. When a Su bBoard is d iscovered , it is treated as a board . Board s w ith u nreachable statu s im p ly a second ary failu re. 79 Fognet’s Field Guide to OpenView NNM APA board statu s event varbind s: Varbind # Description ---------------------------------------------------------------------------------$2 Tim estam p event occurred $3 N am e of the nod e that contains the board $5 Capabilities $6 Managem ent Ad d ress $7 Route Distinguisher $9 Ind ex of the responsible board $11 SubBoard ind ex of the responsible board $13 Serial N umber $14 Mod ule Nam e $15 Mod ule Description $16 Hard ware Version $17 Software Version paConfig.xml: the APA configuration file The location of this file is: $OV_CON F/ nnm et/ p aConfig.xm l. The schem a is d efined in: p aConfigSchem a.xsd . Changes take affect w hen ovet_poll is restarted via ovstart. Rem em ber to backu p p xConfig.xm l file before m aking changes. Cu stom izations need to be m erged in on u p grad es and p atches. p aConfig.xm l cu stom ization and m od ification best p ractices are: 1. Backup the paConfig.xm l file, track revisions for reverting 2. Docum ent netmon and APA configuration custom izations 3. Use ovet_topodump.ovpl to test that the nod es or interfaces pass the class filter created or mod ified 4. Valid ate XML syntax using an xm l ed itor or w eb brow ser 5. Test changes using checkPollCfg in support subd irectory 6. Restart ovet_poll process to begin using the new settings 7. verify w hich filter is used for node and/ or interface w ith ovet_d emand poll -d nod enam e Sim p lified p aConfig.xm l schem a w ith p aram eterList exam p les: <paConfig> <subSystemConfig>PollingEngine,StatusAnalyzer,Talker <globalParameters>statisticsEnable <configGroupList> <configGroup> pollingSettings;configPollSettings <generalParameters> GenerateDegradedEvent <classSpecificParameters> <defaultParameters>interval;snmpEnable;timeout 80 Status Polling - APA <classSpecification> isRouter, isSwitch, <parameterList> interval;snmpEnable;timeout paConfig.xml evaluation order issues Sp ecifications are evalu ated in top to bottom ord er , so the first sp ecification that p assed the filter for a p articu lar d evice is the one that ap p lies. In N N M 7.01, an IF filter w ou ld take p reced ence over the nod e level for statu s sp ecifications if the IF filter ap p eared before the nod e sp ec. In N N M 7.5, the above behavior w as changed so that a statu s setting of false for a nod e m akes that statu s false for every IF on that nod e. For the tru e case, how ever, a false IF setting changes the statu s of the IF on a nod e set to tru e from false. In N N M 7.01, if a filter sp ecifies a statu s for p ing, bu t not SN MP (or vice versa), the d efau lt p aram eter w ou ld be ap p lied to the m issing sp ec and all other sp ecs w ou ld be ignored . In N N M 7.5, m u ltip le filters can be evalu ated and the first filter to sp ecify a sp ec w ill be evalu ated . Only after all sp ecs have been evalu ated and no sp ec for either p ing or SN MP has been fou nd shou ld the d efau lt be ap p lied . Interface ICMP polling Prior to N N M 7.51 Interm ed iate Patch 18, the APA w as hard cod ed to m ark all the IP interfaces as p olled d isabled and for ICMP, and this cou ld not be changed via the p aConfig.xm l file. That p atch introd u ced the isDiscoContrivedIF filter and by setting the pingEnable valu e to tru e in this filter, IP interfaces can be m onitored u sing ICMP. Synchronizing poll settings w ith configuration polls N N M 7.5 introd u ced the configu ration p oll, so if statu s p olling is being d isabled to a nod e or interface in the PollingSettings config grou p , it m ay be a good id ea to d isable p olling in the ConfigPollSettings config grou p as w ell. Configu ration p olls are alw ays SN MP. paConfig.xml polling granularity class specifications Polling granu larity in the APA is d efined by classSp ecifications w hich corresp ond to top ology filters (d iscu ssed below ) that are called w ithin each classSp ecification block. 81 Fognet’s Field Guide to OpenView NNM In p aConfig.xm l, these settings are located u nd er: <subSystemConfig> <name>PollingEngine</name> <configGroup> <name>PollingSettings</name> <classSpecificParameters> <defaultParameters> <classSpecifications> The follow ing class sp ecifications d efine the p olling granu larity for the APA p oller in the p aConfig.xm l file. Details on the m eanings of these sp ecifications can be fou nd in the APA w hite p ap er. The entries below are for N N M 7.51. They are u nchanged from N N M 7.5 excep t for the ad d ition of the isN ew N od e and isN ew Inter face sp ecs. Und er N N M 7.01, there w as a m u ch m ore lim ited set of sp ecifications: Lines in italic below are com m ented in p aConfig.xm l by d efau lt and w ill not be u sed u nless u ncom m ented . Class Specification snm pEnable pingEnable _____________________________________________________________ A PA N oPollNodes false false isN ew Nod e true true isN ew Interface true true isIpPhone false false ifsWithAnycastAdd rs n/ a false isRouter true true AvayaIptDevices true true N otConnected Snm pSw itch n/ a true isSw itch true false isEnd N od e false true W anIf true false IfTypeFilter n/a false isPartOfAggregatedIF true n/ a IFInN otConnected Sw itch n/ a true Unconnected Ad m inUpOrTestRouterIf true true Unconnected Ad m inUpOrTestSw itchIf false false Unconnected Ad m inDownRouterIf false false Unconnected Ad m inDownSw itchIf false false Unconnected End Nod e false true N otConnectedIF false false A llBoards false n/a N oPingA ddresses n/a false Using ET topology filters to specify polling granularity p aConfig.xm l class sp ecifications are d efined u sing extend ed top ology filters. Extend ed top ology filters are a sep arate entity from N N M trad itional filters. To see a list of all existing filters, ru n: ovet_topodump.ovpl –lfilt 82 Status Polling - APA To see a d u m p of d iscovered d evices that p ass a given filter, ru n: ovet_topodump.ovpl -node -filt [filtername] ClassSp ecification filters are evalu ated in xm l file ord er, so if a d evice m atched both isSw itch and isRou ter, the isRou ter ru le w ou ld ap p ly. Extend ed Top ology Filters are d efined in: $OV_CON F/ nnm et/ topology/ filter/ TopoFilters.xm l V7.51 p rovid es a d iscu ssion of ET Top ology Filters w ith exam p les in: $OV_DOC/ ETFilter.pd f In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. Filter assertion type attributes Prior to V7.53, ET Filter assertion typ e attribu tes w ere not d ocu m ented , bu t Kevin Sm ith of H P d ocu m ented them in his N N M 7.51 Dep loym ent hand book w hich can be fou nd at: http:/ / w w w.fognet.com/ NN M_7.51_Deploym ent_H and book_v1.3.pd f Assertions are d ocu m ented in the V7.53 Using_Extend ed _top ology .p d f u ser m anu al bu t are listed here for those ru nning p reviou s versions. Assertions for Interface Containers, H SRP Grou p s and Ad d resses w ere d efined in Kevin‟s d ocu m ent and m ay not have been su p p orted p riot ton V7.53. Use the assertions below to bu ild cu stom ET filters for APA statu s or for Container View s: N ode Assertion Attribute Types Attribute Type Description ---------------------------------------------------------------------------------------------------nam e The internal nam e of the nod e SysN am e SysN am e of the nod e lastUpd ateTim eUTC The last tim e the nod e was updated d escription The d escription of the nod e IPAd d ress Ad d ress on the nod e sysOID Match on SN MP System OID of the node capability The capability of the node status The overall status of the nod e extensibleAttribute An “extensible” attribute of the nod e H ostIDFile Use H ostNam e or IP Ad d ress in the file Card Assertion Attribute Types Attribute Type Description 83 Fognet’s Field Guide to OpenView NNM ---------------------------------------------------------------------------------------------------nam e The nnm entity nam e of the card lastUpd ateTim eUTC The last tim e the card was updated ind ex The card ind ex d escription The d escription of the card type MIB type field on the Card m od el MIB mod el field on the Card sn MIB serial num ber field on the Card fw version MIB firm w are version field on the Card hw version MIB hard ware version field on the Card sw version MIB software version field on the Card com ponentNam e SN MP System OID of the card status The overall status of the card extensibleAttribute The extensible attributes of the card card Adm inStatus Card ad m instration status cardOperStatus Card Operation Status m ibType MIB type from w hich card d ata was read Interface Assertion Attribute Types Attribute Type Description ---------------------------------------------------------------------------------------------------lastUpd ateTim eUTC The last tim e the interface was upd ated . IPAd d ress IPv4 ad d ress bound to the Interface ifDescription The d escription of the interface ifDesc The d escription of the interface ifAlias Interface N am e Alias ifN am e Interface N am e Alias ifInd ex Interface Ind ex vlanPortType The role of this port in VLAN config ifAd m inState Interface ad m instration state ifOperStatus Interface Operation Status ifType Interface Type ifSpeed Interface Speed status The overall status of the interface extensibleAttribute The extensible attributes of the interface capability The capability on the Interface Interface Container Assertion Attribute Types Attribute Type Description ---------------------------------------------------------------------------------------------------nam e The nnm entity nam e of the If container lastUpd ateTim eUTC The last tim e the If Container was upd ated d escription The d escription of the If Container type MIB type field on the If Container status The overall status of the If Container extensibleAttribute The extensible attributes of the If Container HSRP Group Assertion Attribute Types Attribute Type Description ---------------------------------------------------------------------------------------------------virtualAd d ress Virtual Ad d ress of the H SRP Group Address Assertion Attribute Types Attribute Type Description ---------------------------------------------------------------------------------------------------- 84 Status Polling - APA IPAd d ress reachabilityState extensibleAttribute IP Ad d ress state of the ad d ress The extensible attributes of the add ress Filter nodes based on SN MP sysObjectID In this exam p le, nod es w hose SN MP agent m atches Blu ecoat SN MP SysOID 1.3.6.1.4.1.3417.1.1.23 w ill be not p inged . Ad d the follow ing nod e filter to Top oFilters.xm l: <nod eAssertion nam e="isBlueCoat" title="isBlueCoat" d escription="BlueCoat d evices"> <operator oper="N OOP"> <attribute> <sysOID>1.3.6.1.4.1.3417.1.1.23</ sysOID> </ attribu te> </ operator> </ nod eAssertion> N ext ad d an interface assertion filter to m atch all interfaces on these nod es: <interfaceAssertion nam e="IFInBlueCoat" title="IFInBlueCoat" d escription="Interfaces in BlueCoat d evices"> <operator oper="N OOP"> <interfaceAssociation ascType="inN od e">isBlueCoat </ interfaceAssociation> </ operator> </ interfaceAssertion> Valid ate the filters w ith ovet_topodump.ovpl then ad d the follow ing tw o class sp ecifications near the top of p aConfig.xm l file: <classSpecification> <filterN am e>isBlueCoat</ filterN am e> <param eterList> <param eter> <nam e>snm pEnable</ nam e> ... <value>true</ value> ... </ param eter> <param eter> <nam e>pingEnable</ nam e> ... <value>false</ value> ... 85 Fognet’s Field Guide to OpenView NNM </ param eter> </ param eterList> </ classSpecification> <classSpecification> <filterN am e>IFInBlueCoat</ filterN am e> <param eterList> <param eter> <nam e>snm pEnable</ nam e> ... <value>true</ value> ... </ param eter> <param eter> <nam e>pingEnable</ nam e> ... <value>false</ value> ... </ param eter> </ param eterList> </ classSpecification> Ellip ses in the above xm l rep resent sections left ou t that can be cop ied from sim ilar d efinitions. Valid ate changes u sing checkpollcfg –o <IP of node>, then restart ovet_poll and then ru n ovet_demandpoll.ovpl <IP of node>. Force a device to be polled via ICMP or SN MP only To accom p lish this, bu ild a top ology filter and a corresp ond ing entry in p aConfig.xm l, p erhap s by IP Ad d ress. In the p aConfig.xm l file, cop y the configu ration for a d evice that is cu rrently filtering for that nod e, and p lace it above the existing entry in the file. Then m od ify the cop ied configu ration and change the ICMP or SN MP p olling booleans. Test the changes w ith checkPollCfg then restart the ovet_poll d aem on. In ord er to force p olling via ICMP only, a qu ick and d irty alternative is to set an invalid SN MP com m u nity string for the d evice in the SN MP configu ration GUI. N ote that changing SN MP configu ration retries and tim eou ts w on‟t help , thou gh, as the SN M P Configu ration p ollerrelated settings only ap p ly to netmon’s p olling intervals. SN MP tim eou t and retry settings for APA are set in p aConfig.xm l in the ConfigPollSettings configu ration grou p . Only the com m u nity string and SN MP version settings are read in tot he APA from the SN MP Configu ration GUI (xnmsnmpconf). 86 Status Polling - APA Filtering by ifType (APA) Tw o d efau lt filters ship w ith p aConfig.xm l in N N M 7.5 and filters are com m ented ou t by d efau lt: both IfTypeFilter sets ping to false for m atching types; useful to prevent polls from “w aking” ISDN interfaces, etc. W anIf stops APA polling of m atching types in ord er to suppress connection level APA status events W anIf is d efined as w anIfTyp es filter and ed w ith slow IfSp eed s filter, w here slow IfSp eed s inclu d e: 9k, 16k, 56K, 64K. To enable either of these filters, rem ove the com m ents at the end and the beginning of the filter d efinition; then m od ify the d efau lt ifTyp es in Top oFilters.xm l (if d esired ), then ru n: ovstop ovet_poll and ovstart ovet_poll. For interfaces that are u nd esirable, netm on.interfaceN oDiscover is u sed by netm on to p revent d iscovery and it it is also u sed by ET to p revent d iscovery. More on p age 53. A com p lete list of ifTyp es can be fou nd at: w w w.iana.org/ assignm ents/ ianaiftype-m ib D isable ICMP to a firew all (APA) This requ ires establishing a new nod e assertion and new class sp ecification. H ere are the step s: 1. Backup paConfig.xm l and TopoFilters.xm l 2. Determ ine the SN MP sysObjectID of the firew all: 3. N eighbor View , Right Click, Details, or run: snmpget –T <firewallName> system.sysObjectID.0 4. In TopoFilters.xm l, copy and paste an OID-based nod e assertion block 5. Change nam e, title, d escription and OID block to m atch firewall d evice 6. Check xm l syntax and confirm filter m atches the d evices by running: ovet_topodump.ovpl –node –filt <newNodeAssertionName> 7. In paConfig.xm l, copy entire isRouter ClassSpecification 8. Paste before isRouter ClassSpecification 9. Change the ClassSpecification nam e to m atch new nod eAssertion nam e 10. Change pingEnable param eter to false 11. Check xm l syntax and confirm polling settings: $OV_SUPPORT/checkPollCfg –o <firewallName> 87 Fognet’s Field Guide to OpenView NNM 12. Apply changes by running ovstop/ovstart on ovet_poll Sw itching routers and routing sw itches in the APA If a nod e is both a sw itch and a rou ter, som etim e called a sw ou ter, then APA p olls it as a rou ter by d efau lt. That is d u e to the fact that the isRou ter class sp ecification is above the isSw itch class sp ecification in the p aConfig.xm l file. By d efau lt, a sw itch/ rou ter is both SN MP p olled and p inged . It also m eans that Connected and Unconnected Ad m in UP interfaces w ou ld be both SN MP p olled and p inged . With the release of N N M 7.53, the follow ing d ocu m ent inclu d es an ap p end ix w hich w alks throu gh a m ethod ology for hand ling d evices w ith both sw itch and rou ter fu nctionality: $OV_DOC/ w hitepapers/ ETand APADeploym entGuid e.pd f If ru nning versions p rior to 7.53, Kevin Sm ith of H P had w ritten the “N N M 7.51 Dep loym ent H and book” w hich contained the exact sam e m ethod ology and also, coincid entely p erhap s, ap p ears in Ap p end ix A of his d ocu m ent. This hand book can be d ow nload ed from : w w w.fognet.com / NN M_7.51_Deploym ent_Hand book_v1.3.pd f Interesting sw itch interface filter example The follow ing exam p le p rovid ed by N ils Johannessen d em onstrates a typ ical “InterestingSw ou terIF” assertion that is introd u ced in the section above. In this exam p le, the list of interesting interfaces is lim ited to interfaces having IP ad d resses w ithin a range as w ell as those being insid e that range and also lim ited to the set of ifTyp es given: !-- added 2007-09-12 Nils A Johannessen - Manag-E Nordic-> <interfaceAssertion name="InterestingSwouterIF" title="" description=""> <operator oper="OR"> <attribute> <capability>isL2Connected</capability> </attribute> <attribute> <ifType>24</ifType> <!-- 24 = loopback --> </attribute> <attribute> <ifType>53</ifType> <!—- 53, 135,136 = VLAN --> </attribute> <attribute> <ifType>135</ifType> 88 Status Polling - APA </attribute> <attribute> <ifType>136</ifType> </attribute> <attribute> <ifType>131</ifType> <!-- 131 = tunnels --> </attribute> <attribute> <IPAddress> <IPv4> <address>10.150.*.*</address> </IPv4> </IPAddress> </attribute> </operator> </interfaceAssertion> APA topology events The follow ing events are log-only by d efau lt, bu t can be set to log to get variou s d etails abou t APA d ecision m aking. Alternatively, u se the ovdumpevents com m and to view these log-only events: OV_TOPOLOGY_Attr_Change_Notification Issued w henever an ET topology object‟s attribute changes OV_TOPOLOGY_Life_Cycle_N otification Issued w hen an ET object is created or deleted OV_TOPOLOGY_Topology_State_Notification Rich d ata pertaining to the status of ET Discovery OV_TOPOLOGY_Status_Change_N otification Issued for every ET topology status change APA and important node filters Im p ortant nod es are d efined in the MyH ostID.xm l filter file to configu re the APA to alw ays send u ncorrelated alarm s associated w ith d evices to the alarm brow ser. While the references to this filter in the xm l files sp eak of byp assing the sp ecific correlation that id entifies second ary failu res, in fact this filter byp asses all ECS correlations. The file accep ts nod e nam es or IP ad d resses and has good exam p les of w ild card sp ecifications. The m yH ostID.xm l file is called by the Im p ortantN od e filter d efined in the Top oFilters.xm l w hich is called in tu rn by the Im p ortantN od es ClassSp ecification w ithin the p aConfig.xm l file. The file is located in: $OV_CON F/ nnm et/ topology/ filter/ MyH ostID.xm l 89 Fognet’s Field Guide to OpenView NNM There is a know n p roblem w ith this file in N N M 7.53 and earlier w here an error is generated w hen MyH ostID.xm l has too m any entries. H P recom m end s u sing IP w ild card s or IP ad d ress ranges to lim it the nu m ber of entries in this file. After u p d ating this file, stop and restart the ovet_poll p rocess. APA performance improvements APA p olling statistics are available both in log-only events and throu gh the Dynam ic View s “Polling/ Alanysis Su m m ary” tab. The follow ing p erform ance im p rovem ent w as su ggested by H P w ith the release of V7.53: In p aConfig.xm l, com m ent ou t the IFInN otConnected Sw itch filter, then change the isSw itch classSp ecification p ingEnable from “false” to “tru e,” and then change the Unconnected Ad m inUp OrTestSw itchIF classSp ecification p ing setting from “false” to “tru e.” Test the new settings w ith checkp ollcg and restart the ovet_poll d aem on. Another p erform ance im p rovem ent it to increase the APA thread cou nts by increasing the follow ing p aram eters in p aConfig.xm l: PollingEngineThread PoolSize statusAnalyzerThread PoolSize Connected vs unconnected interface APA status Prior to N N M V7.50, the APA calcu lated nod e statu s based on the statu s of all the p olled interfaces w hether connected or not connected in ET. In V7.51 this changed so the APA w ill not p rop agate the failu re of u nconnected interfaces to the nod e‟s statu s. The below p aConfig.xm l flag can be u sed to fine tu ne this new algorithm . If the valu e of this flag is tru e APA w ill p rop agate the statu s of u nconnected interface to the nod e and vice versa. By d efau lt the valu e of this flag is tru e for all the interfaces excep t for all the u nconnected Ethernet interfaces in connected nod es: propagateStatusForUnconnectedObj To revert back to N N M7.50 behavior, search for the follow ing p aConfig.xm l filter and set the flag to tru e. N ote that this m ay cau se container view to show incorrect statu s of netw ork : UnconnectedEthIFInConnectedNode 90 Status Polling - APA APA memory footprint reduction N N M v7.51 introd u ced this enhancem ent w hich only load s actively p olled interfaces into m em ory. By d efau lt, the APA load s all d iscovered interfaces into m em ory. To im p lem ent this im p rovem ent, locate the below p aram eter in the p aConfig.xm l and change its varValu e from false to tru e, then restart ovet_poll: load OnlyPolled ObjectsIntoMem ory N ote that in the V7.53 d ocu m ent listed below , sp ecial consid erations need to be taken into conseration on N N M system s ru nning H P-UX w hen im p lem enting the m em ory footp rint red u ction: $OV_DOC/ w hitepapers/ ETand APADeploym entGuid e.pd f D isabling status bridge Those w ho rely soley on H om ebase view s and are u sing APA can im p rove p erform ance by d isabling the statu s brid ge. This m eans that ovw-based view s w ill not have active statu s. Sim p le set the follow ing p aConfig.xm l p aram eter to false and restart ovet_poll to accom p lish this: StatusBridgeEnabled Characterizing APA polling behavior One w ay to d iscover w hat filters are in effect for a p articu lar set of nod es is to change the p olling interval for a filter to a u niqu e valu e . Look for that u niqu e valu e in the ou tp u t of the checkPollCfg com m and . For d etailed tracking of w hat the APA is d oing, enable the APA Top ology events d escribed in the section im m ed iately above. For exam p le, if the p olling interval for the isRou ter filter w as changed from 300 second s to 299 second s, then those resu lts from the checkPollCfg com m and that show ed a p olling interval of 299 second s w ou ld ind icate the isRou ter filter is the filter that ap p lies to those objects. The “ -V” and “ -d” com m and line op tion s for the ovet_demandpoll com m and p rovid e, resp ectively, internal states of APA objects and d etailed d u m p of the active d em and p oll. Improving SN MP-based status w ith the APA Defau lt APA behavior is to fail an interface that r etu rns an SN MP noSu chObject error if the SN MP agent re-ind exes or looses track of a 91 Fognet’s Field Guide to OpenView NNM p articu lar interface. V7.51 introd u ced a p aram eter w hich forces the APA to p oll su ch an interface via ICMP before issu ing a failu re event. To im p lem ent this im p rovem ent, locate the below p aram eter in the p aConfig.xm l and change its varValu e from false t o tru e, then restart ovet_poll: useIcmpIfSnmpNoSuchObj APA and ManagementAddesss picking SN MP p referred m anagem ent ad d resses are initially d efined throu gh netmon d iscovery, bu t w hen an SN MP m anagem ent ad d ress su bsequ ently fails to resp ond to SN MP p olls, by d efau lt, APA p icks a new m anagem ent ad d ress. This new ad d ress becom es the ad d ress u sed by Extend ed Top ology as w ell. If loop back ad d resses are enabled for m ost of the netw ork d evices in the m anaged environm ent, it is not necessary for the m anagem ent station to p ick a new ad d ress since m ost d evices w ill re -assign the loop back to another w orking interface if the cu rrent interface fails. For this reason, H P recom m end s d isabling the p ickMana gem entAd d r featu re if loop backs are w id ely u sed in the environm ent. To d o this, for V7.01 throu gh V7.51, change the p aConfig.xm l file p aram eter MgmtAddrInhibited from ―false‖ to ―true‖ and restart ovet_p oll. V7.53 introd u ced a new p aConfig.xm l p aram eter called DisablePickMgmtAddress, and setting this is set to ―false‖ by d efau lt. Also, consid er the netmon lrf setting -k adjustNodeSnmpAddr=false w hich is d iscu ssed in m ore d etail on p age 43. If loop backs are not w id ely u sed , there are three op tional p aConfig.xm l p aram eters introd u ces w ith N N M 7.51 that allow the APA p ickManagem entAd d ress to be m ore finely tu ned : • Mgm tAd d rPreferred – For a given ad d ress, if this param eter is true, the ad d ress w ill be consid ered for a new m anagem ent ad d ress over an ad d ress w here this param eter is false as a m anagem ent ad d ress if the current m anagem ent ad d ress fails. In ad d ition, this ad d ress w ill be consid ered even if it is currently not configured to be polled . By d efault, this param eter is set to false. • Mgm tAd d rInhibited – For a given add ress, if this param eter is true, the ad d ress w ill never be used as a managem ent ad d ress if the current m anagem ent ad d ress fails. By d efault, this param eter is set to false. 92 Status Polling - APA • Mgm tAd d rMaxSnm pQueries – When the current managem ent ad d ress fails to respond to SN MP, the pickMgm tAd d ress algorithm w ill engage. This w ill cause no m ore than Mgm tAd d rMaxSnm pQueries sim ultaneous SN MP queries to be issued d uring the search for a new managem ent ad d ress. fails. By d efault, this param eter is set to 10. To get even m ore control, u se DN S to tell N N M w hich ad d ress to u se to as the p referred SN MP ad d ress. See p age 26 and sections su bsequ ent to that. Troubleshooting APA To d eterm ine a d evice‟s p olling settings, ru n: $OV_SUPPORT/NM/checkPollCfg –o <object name> To p rint a su m m ary of objects su bject to p olling, ru n: $OV_SUPPORT/NM/checkPollCfg –l The m ost com m on APA p olling p roblem s relate to m ap p ing of isSw itch and isRou ter. isSw itch and isRou ter flags are assigned by netmon d u ring d iscovery and these cap abilities are u sed by ET top ology filters. Ru n throu gh these checks if certain d evices are not being p rop erly p olled by the APA: 1. Check Firew alls for SN MP/ ICMP blocking. 2. Force isRouter w ith G flag; isSw itch w ith B flag in oid _to_type file (see page 7) 3. Brow se SN MP MIBs; check for cut tables that may prevent visibility to: .1.3.6.1.2.1.1 .1.3.6.1.2.1.17.1 .1.3.6.1.2.1.4.20.1 .1.3.6.1.2.1.2.1 .1.3.6.1.2.1.31.1.1.1.1 .1.3.6.1.2.1.31.1.1.1.18 .1.3.6.1.2.1.4.1.0 .1.3.6.1.2.1.4.21 .1.3.6.1.2.1.4.22.1 .1.3.6.1.2.1.3.1. systemTable for oid_to_type dot1dBaseTable, isSwitch ipAdEntTable ifTable, isRouter/isSwitch ifName ifAlias ipForwardingTable, isRouter ipRouteTable, isRouter ipNetToMediaTable, isSwitch atTable Sam p le qu ery: snmpwalk <target> 1.3.6.1.2.1.1 93 8. Traps, Events and Alarms This section covers som e basics abou t N N M‟s event infrastru ctu re. More d etails abou t ind ivid u al events are covered in the follow ing section, Interp reting Events. What is a trap vs. an event vs. an alarm? In short, a trap becom es an event, and then an event becom es an alarm . “Event” is the generic term u sed in H P‟s d ocu m entation for w hat m ost often is technically an alarm . An SN MP trap is an u nsolicited , u nacknow led ged notification sen t from an SN MP agent to an SN MP Manager. N N M w rap s SN MP trap s w ithin the OV_EVEN T stack of the pmd d aem on. This ad d s attribu tes to the SN MP trap that are not otherw ise d efined u nd er SN MP, su ch as severity, logging behavior, and event category. Au tom atic actions are also ad d ed -valu e attribu tes to m ake the trap and event, and these attribu tes are ad d ed throu gh the event configu ration GUI w hich is a front-end to the trap d .conf configu ration file. A trap or an event p asses into pmd, and OV events com e ou t – they are no longer SN MP trap s. ovtrapd is the d aem on resp onsible for receiving trap s on UDP p ort 162 by d efau lt and bu ffering them for pmd. An alarm is sim p ly a rep resentation of an OV event in the alarm brow ser. Attribu tes that alarm s have inclu d e acknow led gem ents, d eletions, event correlation cou nts and em bed d ed alarm relations. When an alarm is d eleted from an N N M u ser‟s alarm brow ser, the u nd erlying N N M event rem ains in the eventdb. Configuring SN MP traps via trap macros RFC 1215 d efines the TRAP-TYPE m acro and RFC 2578 d efines the N OTIFICATION -TYPE m acro. These m acros d efine a w ay of su p p lying the trap d efinitions that d evices m ay u se w ithin a MIB d efinition file. Essentially, a TRAP-TYPE m acro is an SN MPv1 trap , 94 Traps, Events and Alarms w hile a N OTIFICATION -TYPE trap / notification. m acro d efines an SN MPv2 When load ing MIB files u sing the load MIB m enu op tion w ithin the ovw GUI (or by ru nning: xnmloadmib), N N M d etects m acros and p rom p ts the u ser for confirm ation to u p load em bed d ed trap d efinitions if they exist. If, w hen load ing a MIB, no su ch m essage is d isp layed for the u ser, the MIB d oes not contain em bed d ed trap d efinitions. The trap d efinitions can be m anu ally extracted from the MIB files w ithou t u p load ing the MIBs. Becau se load ed MIBs consu m e m em ory, this m ay im p rove N N M‟s scalability in environm ents w here m em ory is at a p rem iu m . To ju st load the trap d efinitions w ithou t load ing the MIBs, ru n : xnmloadmib -event -trapType -trapDetail 0 –load <MIBfile> N ote that if the MIB load u tility is invoked from the com m and line to load a MIB (xnmloadmib –load <file>), it d oes not give the u ser the op tion of im p orting trap d efinitions into trap d .conf file. Ru nning xnmloadmib w ith no op tions to invoke the GUI, how ever, w ill load the trap d efinitions. Configuring events or traps via trap definition files $OV_CONF/C/trapd.conf is the configu ration file that hold s the trap d efinitions. Som e vend ors p rovid e text files that hold N N M trap d .conf com p atible d efinitions that can be u p load ed d irectly into N N M‟s event configu ration. Often, the nam es of these files have the w ord “trap d ” in their filenam es. The com m and s to d o this are: xnmevents –load <filename> xnmevents –replace <filename> xnmevents –merge <filename> See the m an/ ref p ages to d eterm ine w hich com m and is best for the environm ent. The m ost com m on issu e w ith load ing trap d .confcom p atible files is file com p atibility errors d u e to the file form at. Often, assu ring the very first line of the trap d .conf-com p atible file read s “VERSION 3” resolves this issu e. Configuring events or traps manually The third m ethod to load N N M trap d efinitions is m anu ally, via the N N M event configu ration GUI. Follow these step s to d o this: 95 Fognet’s Field Guide to OpenView NNM 1. Look at the enterprise ID. If there is not a d efined enterprise m atching the trap d efinition to be created , create a new enterprise ID using Ed it > Enterprises > N ew . Som e nam e m ust be given to the new enterprise. In som e cases, and OID Alias m ay be specified . See the trap d .conf man/ ref for m ore on that 2. Select the new ly created enterprise then add a specific event that m atches the specific num ber using Ed it > Events > N ew 3. Select the “event message” tab and select the d esired logging behavior, i.e. “Don't log or d isplay” or log to a particular event category 4. If logging the event, d efine a severity and Event Log m essage. See page 99 for a list of variable bind ing variables that can be passed from the trap to the log m essage 5. Optionally test the new event configuration by form atting an e vent using the snmpnotify comm and or the script: $OV_CONTRIB/NNM/sendMsg/sendMsg.ovpl For m ore d etails on m anu ally creating events, see the m an/ ref p age for trap d .conf, and associated control com m and s xnmtrap and xnmevent. Alw ays m ake a backu p of the $OV_CON F/ C/ trap d .conf file w hen m aking event cu stom izations. N ote that d irect ed its to the trap d .conf are not su p p orted by H P, bu t su ch ed its (w hen carefu lly hand led ) can be a p ow erfu l w ay to bu lk-m od ify w hole classes of event d efinitions. When m aking m anu al ed its to the trap d .conf file, they w ill not be activated in the event stack u ntil one of the follow ing com m and s is issu ed : xnmevents –event xnmtrap -event D rop SN MP traps from particular devices This featu re ap p eared in V7.01. In V7.51, the ability to au tom atically u nblock trap s once the flood su bsid es w as ad d ed . In Interm ed iate Patch 18, the “-c” op tion w as ad d ed to allow a blocking tim e interval. Also in V7.51, Som e d ocu m entation for this featu re w ith exam p les ap p eared in: $OV_DOC/ WhitePapers/ EventRed uction.pd f The $OV_CON F/ ovtrap d .conf file is a list of sp ecific IP Ad d resses and trap OIDs that the ovtrapd d aem on w ill reject SN MP trap s from . The file su p p orts w ild card s p er the w hite p ap er, thou gh the ovtrap d .conf m an/ ref p age says that w ild card s are not su p p orted . 96 Traps, Events and Alarms Im p ortant: If u sing the APA p oller, certain trap s are p assed to the p oller to im p rove statu s intelligence. Do not d rop all trap s from d evices that are of concern w ith resp ect to statu s. See the section below for the list of trap s the APA u ses for statu s. If the –B, –b op tion is set (w ith or w ithou t the –r op tion), trap s w ou ld be su p p ressed if a storm occu rs accord ing to these ru les: With the –B op tion, If 1500 events com e in from a d evice at a rate greater than 15 events p er second , block the d evice and d iscard those events. After su p p ression starts, m onitor that ad d ress it for the next 1500 events. If the rate is still greater than 15 events p er second , d iscard the events for another interval, otherw ise u nblock the d evice and allow the events to com e in. Use the –b op tion to change the d efau lt valu e of 1500 events. Use the –r op tion to change the d efau lt valu e of 15 events p er second . The –c op tion enables configu ring a tim e p eriod in second s for the sp ecified blocking criteria. If the nu m ber of trap s com ing from a p articu lar d evice reaches the valu e configu red by "-b" or "-B" in less than tim e d u ration sp ecified by "-c", the trap s w ill be blocked from the d evice. This op tion cannot be u sed w ith "-r" op tion. Once the ovtrap d .conf file is u p d ated , the ovtrap d d aem on m u st be restarted u sing ovstop and ovstart. If ru nning u nd er the UN IX, the d aem on can be issu ed a SIGIN T instead of a restart as follow s: ps -ef|grep ovtrapd|awk '{print $2}'|xargs kill -2 Certain trap s shou ld not be d rop p ed u sing the p roced u re above if p olling d evices u sing the APA, since they are p assed to the p oller t o help d erterm ine d evice statu s. See p age 74 for a list of these trap s. Automatically suppress SN MP trap storms This featu re seem s to have ap p eared in V7.01 or a p atch to 7.01. The – B, -b, and -r LRF op tion s to ovtrapd d isable SN MP trap s from a d evice in the au tom atically w hen a trap storm occu rs. These op tions cau se au tom atic entries to be created in the $OV_CON F/ ovtrap d .conf file. The –B op tion su p p resses trap s if 1500 are received in a 15 second p eriod . The –b and –r op tions allow the granu larity of the nu m ber and rate of trap s to be exp licitly set for su p p ression. For m ore inform ation, see the ovtrapd m an/ ref and see p age 5 for the LRF u p d ate p roced u re. H ere is an exam p le ovtrap d .lrf file: 97 Fognet’s Field Guide to OpenView NNM OVs_YES_START:pmd:-W -b 2000 -r 30:OVs_WELL_BEHAVED:: N ote that the –W is requ ired on Wind ow s bu t not requ ired on UN IX. Generating ad-hoc SN MP traps or N N M events snmpnotify is the facility for generating raw SN MP trap s that can be sent to any SN MP m anager. Later versions of N N M d o not inclu d e the snmptrap com m and , w hich is snmpnotify‘s m ore fam iliar p red ecessor. snmpnotify ou tp u t m ay either be an acknow led ged SN MPv2C inform or an u nacknow led ged SN MPv1 or SN MPv2 Trap . Another m ethod for generating events on the N N M server is to u se the ovevent com m and . This com m and p rovid es N N M-sp ecific d ata not native to SN MP trap s su ch as severity and alarm category. Both ovevent and snmpnotify have sim ilar syntax an d p rovid e the “-d ” op tion for d u m p ing ASN .1 d ecod es. By d efau lt, both s nmpnotify and ovevent p ick u p d ata associated w ith the sp ecified target from the SN MP configu ration d atabase to p ass to the com m and su ch as com m u nity strings and tim eou ts. The “-d ” op tion is u sefu l in trou bleshooting SN MP configu ration issu es. Exam p les of the ovevent com m and can be seen the follow ing ovp l scrip ts in $OV_CON TRIB/ N N M: p op u p Msg, ringBell, send Msg, and setStatu s. D ifferences betw een snmpnotify and ovevent snmpnotify u ses SN MP as a the transp ort w her eas ovevent u ses TCP, w hich som e m ay consid er “m ore reliable.” snmpnotify can be u sed to generate a trap and it can be sent to any SN MP m anager. ovevent form ats N N M events w hich can only be received by other cop ies of N N M. Trap s send u sing snmpnotify alw ays flow into ECS, so can be correlated . Events com ing from ovevent are su bject to the pmd ov_event stack settings in p m d .lrf, so their flow eithe r throu gh or arou nd ECS can be controlled . See the pnum stack op tion in the m an/ ref for ov_event for m ore abou t ECS flow control and see p age 94 for m ore inform ation on the d ifferences betw een trap s and events, 98 Traps, Events and Alarms Event variable bindings The Event Log Message, Pop -u p Wind ow Message, and Com m and for Au tom atic Action field s in the Mod ify Events and Cop y Events d ialog boxes u se sp ecial $ variables to p resent d ata th at w ere received w ith the event. These sp ecial characters can help p rovid e form atted ou tp u t. All nonp rintable characters are converted to their octal (\ 000) equ ivalent for d isp lay in the event brow ser, or w hen p assed to the op erator initiated (m anu al) actions. The tw o excep tion s are that a tab is d isp layed as \ t in the alarm brow ser and as sp aces in p op -u p m essages. A new line is d isp layed as \ n in the alarm brow ser and as a new line in p op -u p m essages. All nonp rintable characters are p assed u nconverted to au tom atic actions execu ted by ovactiond. Sp ecial Characters in action callbacks: a b f n r t v \ \\ 000 Xhh Alert (bell) character Backspace Form feed N ew line Carriage return Horizontal tab Vertical tab Backslash Use to separate elem ents in a pathnam e Octal num ber, ranging from 000 to 177 Hex num ber, both hh characters m ust be 0-9a-fA-F Variables in action callbacks: $1 $# $* $-n $+n $>n $>-n $>+n $x $X $@ $O $o $V $r $ar The first sequential attribute of the event (varbind ) The num ber of attributes in the event All attributes as: seq num , nam e (type): value strings The nth attribute as: seq num , nam e (type): value string The nth attribute as: nam e: value string All attributes greater than n as value strings. $>0 = $* All attributes greater than n as seq nam e (type): value All variables greater than n as nam e: value strings Date event received using local d ate representation Tim e event received using local tim e representation Epoch tim e (second s since Jan 1, 1970) using tim e_t The nam e (object id entifier) of the receiv ed event The (object id entifier) as a string of num bers Event type (SN MPv1, SN MPv2C, CMIP, GEN ERIC) The im plied "source" of the event in textual form . This m ay not be the "true source" if proxied . See $R below Sam e as $r except print the source as an IP add ress 99 Fognet’s Field Guide to OpenView NNM $R $aR $c $s $N $F $U $$ $C $E $O $e $A $aA $G $S $T The "true source" of the event in textual form . If the event w as forward ed , this d isplays the ad d ress of the rem ote pmd' s machine Sam e as $R except print the source as an IP ad d ress The category the event belongs in The severity of the event The nam e (textual alias) as d efined in trapd.conf The textual nam e of remote pm d 's m achine if this event w as forward ed , else local machine's nam e The NN M UUID of the event as a string of num bers Print the $ character The trap com m unity string The trap enterprise as text string from trapd .conf The trap enterprise as text string from Load ed MIBs The trap enterprise as Object ID string of num bers The trap agent ad d ress as d efined in the trap PDU If the nam e server can resolve, print the nod e nam e Sam e as $A except print the source as an IP ad d ress The trap's generic-trap num ber The trap's specific-trap num ber The trap's sysUpTim e tim e-stam p. This is the rem ote m achine's tim e in hund redths of a second between the last re-initialization of the d evice and the generation of the trap. For non-SN MPv1 events this value is 0 Event logging Events are logged to the event d atabase. Prior to N N M 6.0, events w ere logged to a flat file called trap d .log. The p referred m ethod for accessing events ou tsid e the alarm brow ser is to u se the ovdumpevents com m and . Pre-N N M 6.0 behavior, how ever, can be restored by m od ifying the pmd LRF file and ad d ing the follow ing sw itch: -SOV_EVENT;t;l8 This logs events to $OV_LOG/ trap d .log and sets the logfile to roll at 8MB. See p age 5 for LRF u p d ate p roced u re. N ote that there is a significant p erform ance hit w ith logging events to trap d .log. There are three general logging m od es: IGN ORE, w hich d iscard s the event entirely and is configu red via the “Don‟t log or d isp lay” op tion in the GUI; LOGON LY, w hich send s the event to the event d ataba se bu t d oes not send it to the alarm brow ser; and Alarm Categories, w hich log the events to the u ser -cu stom izable set of categories w hich can be configu red via Ed it->Alarm Categories in the event configu ration GUI. 100 Traps, Events and Alarms Som etim es, it is d esirable to log all ignored SN MP trap s for trou bleshooting. N ote once again that m ost of these m ay be Op enView enterp rise alarm s. While not su p p orted or necessarily recom m end ed , globally searching and rep lacing the IGN ORE flag w ith the LOGON LY flag can be easily d one in Wind ow s and UN IX text ed itors. Alw ays m ake backu p cop ies of trap d .conf before ed iting d irectly. In UN IX, u se sed: cp trapd.conf trapd.orig sed '/^EVENT/s/"IGNORE"/"LOGONLY"/' trapd.orig > trapd.conf xnmtrap -events D umping the entire ev ent db Use the below com m and s to d u m p the raw event d atabase. The m ain reason for d oing this is to view events that m ay have fallen ou t of the alarm brow ser becau se the alarm brow ser is lim ited in the nu m ber of events it d isp lays. Also, “log-only” events can only be d isp layed u sing these com m and s: ovdumpevents ovdumpevents –t Dum p entire eventdb to std out Dum p entire eventdb to std out, then tail output (Ctrl-C to stop ) These com m and s w ill show a su m m ary of regu lar and correlated events. N ote that ovdumpevents can be CPU-intensive and that the p rocessCorrEvents tool is UN IX Only ovdumpevents –s ―default‖ > event.log $OV_SUPPORT/processEvents event.log event-summary ovdumpevents –c ―default‖ > corr.log $OV_SUPPORT/processCorrEvents corr.log corr-summary Interpreting ovdumpevents output Below is an exam p le of an event from ovdumpevents ou tp u t: 1162047602 1 Sat Oct 28 11:00:02 2006 cloud y.fognet.com N IF lan0 Up Capabilities: isN od e Root Cause: cloud y.fognet.com lan0;1 17.1.0.58916866 84819 Each event in ovdumpevents ou tp u t is a single line of the form : Tim eStam p Cat Tim e EventSrc SWSrc EventMsg ; Sev EventOID OVObjId Tim eStam p is the Ep och Tim e that the event w as received . 101 Fognet’s Field Guide to OpenView NNM Cat is the nu m eric category as d efined in trap d .conf. In the exam p le above, it is “1” w hich is “LOGON LY.” The d efau lt catego ries are: CATEGORY 0 "IGN ORE" CATEGORY 1 "LOGON LY" " CATEGORY 2 "Error Alarm s" CATEGORY 3 "Threshold Alarm s" CATEGORY 4 "Status Alarm s" CATEGORY 5 "Configuration Alarm s" CATEGORY 6 "Application Alert Alarm s" CATEGORY 7 "Problem Diagnosis Alarm s" Tim e is the hu m an-read able translation of the Ep och Tim e. EventSrc is the nod e that p rod u ced the event. N ote that for statu s events w hose origin is actu ally the Op enView server, the EventSrc w ill be set to the target of the event. In the above exam p le, the EventSrc is “clou d y,” w hich is the nod e Op enView d etected as having Interface Lan0 com e u p , bu t the nod e that actu ally p rod u ced the event w as “p atchy,” the m anagem ent server. Use the flag that follow s, SWSrc, to help d eterm ine the tru e sou rce of Op enView events. SWSrc is a single character rep resenting the softw are sou rce of the event. If the sou rce is “-“ then the event sou rce is an SN MP trap generated by a sou rce other than Op enView itself. In the exam p le above, SWSrc is “N ,” w hich is netmon, so therefore the actu al sou rce of the event w as Op enView . The p ossible valu es for SWSrc are: C: xnmcollect D: snm pCollect E: xnm events F: ovtopofix I: ipmap J: ovalarm srv L: xnm load m ib M: ovtopmd N : netmon P: pm d R: ovrepld T: ovtrapd b: nnm_brid ge c: xnm topoconf a: Generic application d : nmd em and poll e: ECS Engine i: ECS Circuit l: loadhosts m : netmon m ask change n: xnm polling o: ovactiond p: ovspm d / ovpause / ovresum e r: rem ote pm d s: xnm snm pconf t: xnm trap 6: IPV6 Polling Agent -: Default ?: N one of the above EventMsg is the event log m essage text. Severity is a single nu m ber as follow s: N ormal 102 1 Traps, Events and Alarms Warning Minor Major Critical 2 3 4 5 EventOID is the SN MP Object Id enfier of the event. OvObjId is the Op enView Object ID, or 0 if not available. D umping parts of ev ent db Use the -l (ell) op tion to the ovdumpevents com m and to d u m p events betw een the nu m ber of m inu tes sp ecified and the tim e the com m and is ru n. For exam p le, to d u m p all events logged in the last d ay, ru n: ovdumpevents -l 1440 The follow ing scrip t p rovid es a tem p late that can be cu stom ized to select sp ecific events from the eventdb. By d efau lt, the scrip t d u m p s events generated “tod ay” into files sep arated by severity, e.g. severity.Critical.qry.ou t: $OV_CON TRIB/ NN M/ event/ EventsBySeverity.ovpl Ad hoc queries of the ev ent db ovdwquery allow s d irect access to the d ata w arehou se via SQL. The follow ing exam p le ou tp u ts all events received tod ay: ovdwquery -u ovdb -password –ovdb –file a.qry The file a.qry‟s contents m ight look like this: select message from nnm_event_detail where $BEGIN_TODAY <= event_timestamp and event_timestamp < $NOW; Du m p nod es from top ology: echo ―select ip_hostname from nnm_nodes;‖ |ovdwquery Trap and event forw arding Forw ard ing raw SN MP trap s to other SN MP m anagers is p roblem atic and u nsu p p orted by N N M. N N M events m ay be forw ard ed to other cop ies of N N M, how ever. Ind ivid u al events can be forw ard ed throu gh the event configu ration GUI (xnmtrap). Forw ard ing all events is p roblem atic, bu t a scrip t to insert the p rop er forw ard ing d ata into the record for every logged event d efined in trap d .conf is not too d ifficu lt 103 Fognet’s Field Guide to OpenView NNM to w rite (m ake a backu p cop y of trap d .conf first). The p roblem is that trap s su bsequ ently ad d ed via m acros in MIBs w ill not have the forw ard ing inform ation ad d ed au tom atically. N ote that N N Mforw ard ed events u se TCP p ort 162, not UDP p ort 162, and only cop ies of N N M, OVO or IBM‟s Tivoli N etView can receive them . An alternative to u sing N N M‟s event forw ard ing facility is to set u p an au tom ated action that creates an SN MP trap u sing the snmptrap or the snmpnotify com m and . Another alternative is to u se N N M‟s SN MP API or CSOV to bu ild an ap p lication that cop ies the event stream and generates SN MP trap s to send to the target ap p lication. See p age 174 on u sing N N M‟s APIs or CSOV. Many third p arty p rod u cts exist that can forw ard raw SN MP trap s or N N M events. Tavve (w w w .tavve.com ) p rovid es a free toolkit that is a p op u lar m ethod for bu lk-forw ard ing N N M events as SN MP trap s to rem ote d estinations. Bytesp here‟s SN MP Trap Manager is another free tool that can m anage SN MP trap s forw ard ing, and is available from the follow ing u rl: w w w.oid view.com / trap_fault_managem ent.htm l Op enView Op erations p rod u ct has an agent-based trap receiver w hich is cap able of receiving trap s from N N M in the ov_event form at from the pmd d aem on d irectly. OVO SN MP trap tem p lates are then u sed to filter the feed of all m essages from N N M to OVO. When forw ard ing events from N N M ru nning u nd er Wind ow s, there is a know n p roblem in that the forw ard ed events d o not have the original event‟s AGEN T-ADDR p rop erly encod ed in the forw ard ed trap . Search for H P Dou cm ent ID OV-EN 000876 for d etails on the w orkarou nd , w hich is basically to set the –W LRF sw itch for the ovtrapd d aem on. More on setting LRF sw itches on p age 5. OpenView enterprise and N N M-generated events Events generated u nd er the Op enView enterp rise are those generated by N N M itself. For Op enView generated events that concern a m anaged nod e, su ch as statu s events, the sou rce of the event is alw ays u sed as the sou rce of the event in the alarm brow ser, even thou gh the tru e sou rce is the N N M server itself. This is carried throu gh the sp ecial varbind .1.3.6.1.4.1.11.2.17.2.2.0. Most N N M statu s events also carry the sou rce as regu lar varbind . For netmonbased statu s events it‟s $2 and for APA-based statu s events it‟s $3. 104 Traps, Events and Alarms N N M also u ses events for inter-p rocess com m u nication. Most of these events are configu red to “d on‟t log or d isp lay ” (ignore), bu t they can u sefu l for trou bleshooting N N M and for other p u rp oses. In event th d u m p s, the 5 attribu te is the Op enView d aem on that generated the event. The OV_EVEN T m an/ ref show s w hat d aem ons corresp ond to w hat cod e letter in the d u m p , w here M is ovtopmd, N is netmon, etc. This is instru ctive in interp reting events. All external trap s shou ld show ovtrapd (T) as the sou rce. It is also instru ctive to note that netmon-based interface statu s events are from netmon, bu t nod e-level events are from ovtopmd. Und erstand ing that nod e statu s events are generated as the resu lt of a set of top ology cond itions rather than from actu al statu s is very im p ortant. A nod e u p event is generated “w hen all interfaces are u p ” and d oesn‟t alw ays corresp ond to a nod e d ow n event and vice versa. netmon p rovid es statu s for interface-level entities only. The APA, on the other hand , p rovid es d irect statu s at fou r sep arate levels. Alarm and icon status color defaults Color d efau lts can be changed w ithin $APP_DEFS/ OVw . See p age 9 for m ore on $APP_DEFS. Op erational Statu s Colors: Blue Green Cyan Yellow Orange Red Unknown N ormal/ Up Warning Minor/ Marginal Major Critical/ Dow n Ad m inistrative Statu s Colors: Beige/ Off White Tan Salm on Dark Brow n Unm anged Testing Restricted Disabled Meaning of an Acknow led ged alarm : - Alarm show s as acknow led ged (Checked ) in all brow sers - Alarm does not change status color propagation to category Meaning of a Deleted alarm : - Alarm show s as d eleted (removed ) in all brow sers - Alarm changes status color propagation to category - Und erlying event is not d eleted from eventdb 105 Fognet’s Field Guide to OpenView NNM ov alarmsrv LRF settings Perform ance tu ning p aram eters: -a <num> -d <num> -s <secs> Max events to hold in ovalarm srv internal cache N um . of alarm s to d elete after max alarm s reached H ow often (in second s) to save brow ser state info User control p aram eters: -Baucsd -Ba -Bu -Bc -Bs -Bd -BX -BA Default user control behavior User can acknow led ge alarm s User can un-acknow led ge alarm s User can change an alarm‟s category User can change an alarm‟s severity User can d elete an alarm Exclusive: User has none of these capabilities Exclusive: User has all capabilities (-Baucsd ) Alarm brow ser settings (XN mevent app-defaults) To m od ify alarm brow ser ap p -d efau lts settings: UN IX: Wind ow s: Ed it or add the keyword in $APP_DEFS/ XN mevents. Ed it the follow ing registry key: MKEY_LOCAL_MACH IN E:SOFTWARE\ H ew lettPackard \ OpenView \ N etw ork N od e Manager\ xnm events Som e of the m ore u sefu l of th e settings inclu d e: m axEvents: w arnOnDelete: m axDisplayMsgs: filterByMap: read Only: N um . alarm s viewable in brow ser (3500) “false” turns off confirm ation on deletes Max N um . pop ups to d isplay (unset) Display events only for nod es in m ap (false) Disable any/ all –Bauscd settings (false) Additional actions Ad d itional actions are available from the alarm brow ser m enu bar. These can be extend ed and configu red . To Configu re Actions: Use the com m and s Actions: Add itional Actions to configure actions that are registered through the LRF process (page 5). To lau nch cu stom URL‟s from Actions: View s: Ed it and add URL‟s in xnm eventsExt.conf, close alarm brow ser, run: ovstop ovalarmsrv; ovstart ovalarmsrv, Open alarm brow ser. 106 9. Interpreting Events SN MP agents are typ ically very qu iet by d efau lt in term s of trap generation. Most events that are typ ically seen in the N N M alarm brow ser are generated by N N M itself. These are the Op enView enterp rise events. Mostly, these are N N M‟s p oller resu lts. N ote that all p oller resu lts in the alarm brow ser s how the nod e affected as the sou rce, bu t all p oll resu lts are in fact generated by the Op enView server itself. This can be m islead ing as there is no w ay to know that the actu al sou rce of the event w as the Op enView server versu s the nod e sou rce d isp layed in the alarm brow ser. N N M’s handling of generic and specific traps The SN MPv1 p rotocol d efines seven Generic SN MP trap s as follow s: 0 1 2 3 4 5 6 Cold Start Warm Start Link Up Link Dow n EGP N eighbor Loss Authentication Failure Enterprise The Enterp rise generic is sp ecial becau se it allow s for the u se of the “sp ecific” trap field to accom m od ate vend or extensions via variable bind ings (varbind s), so the Op enView Enterp rise N od e_d ow n event, w hich is generated by N N M, is conveyed w ith the follow ing SN MP d ata: Enterprise: Generic: Specific: Varbind s: .1.3.6.1.4.1.11.2.17.1 6 58916865 1,2,3…n Generic traps and origin event OID s SN MPv2 introd u ced the concep t of an event OID, so generic trap s (like au thentication failu res or link u p / d ow n trap s) w ou ld fall u nd er enterp rise .1.3.6.1.6.3.1.1.5.1 in the MIB tree. The old m ethod w as for 107 Fognet’s Field Guide to OpenView NNM vend ors to d efine p rivate versions of the generics w ithin their enterp rise MIB su btrees. In ord er to accom m od ate the new hand ling of generics w ith the old m ethod , N N M au tom atically p rep end s the trap ‟s enterp rise follow ed by the generic trap nu m ber + 1 to the enterp rise w henever an agent send s a generic trap along w ith a p rivate enterp rise. This w as set u p this w ay so a link d ow n from a Cisco rou ter, for exam p le cou ld be d ifferentiated from a link d ow n from a N ortel rou ter. After N N M v5, the event configu ration GUI lists generic trap s u nd er the .1.3.6.1.6.3.1.1.5 enterp rise. Previou sly, link d ow n events, for exam p le, w ou ld have been listed u nd er generic 3. So a Cisco Link Dow n, for exam p le, w ou ld be configu red in the event configu ration (trap .conf) w ith an id of: .1.3.6.1.6.3.1.1.5.3.1.3.6.1.4.1.9 Taking this ap art, this is the SN MPv2 snm p Trap s enterp rise (.1.3.6.1.6.3.1.1.5) follow ed by the generic trap nu m ber (3) and then the Cisco enterp rise OID (.1.3.6.1.4.1.9.) The alarm brow ser is cod ed to d isp lay any su ch an alarm as a generic link d ow n, i.e. as being from .1.3.6.1.6.3.1.1.5.3. This gives N N M u sers the flexibility to create cu stom event d efinitions m ore generically for w hat w as form erly a m onolithic generic event. So, for any d istinct SN MP agent OID, a d istinct event d efinition can be created . For exam p le, link d ow n trap s for Catalyst 3550‟s w ou ld have the OID: .1.3.6.1.6.3.1.1.5.3.1.3.6.1.4.1.9.1.431. Sim ilarly, the d efinitions can be m ad e higher u p the OID tree. For exam p le, a cu stom event d efinition can be m ad e for link d ow ns for all Cisco OIDs (1.3.6.1.6.3.1.1.5.3.1.3.6.1.4.1.9), or all link d ow n events from all enterp rises (.1.3.6.1.6.3.1.1.5.3). The OID that w ou ld cover all generic events from all enterp rises w ou ld be .1.3.6.1.6.3.1.1.5. N ote that all generic trap s u nd er enterp rise .1.3.6.1.6.3.1.1.5 are set to LOG-ON LY by d efau lt, so none of the trap s d escribed above w ill be seen in the event brow ser u nless they are exp licitly set to log. “Received event X, N o format in trapd.conf” This d efau lt event is generated by H P Op enView and it rep resents an SN MP trap that has been received for w hich the N N M event system 108 Interpreting Events has no translation d efined in N N M. It is cap tu red by the d efau lt enterp rise OID .1.3.6.1.4.1. Events that are cau ght by this trap d efinition need to have event configu rations ad d ed for them u sing one of the three m ethod s d escribed at the begin ning of this section . Bu t before sim p ly d efining an enterp rise to accom m od ate it, it is im p ortant to d eterm ine if the trap is m eaningfu l and d esirable in the first p lace. First, d eterm ine the trap sou rce. Look at the sou rce‟s nam e or IP. Is it in the N N M Top ology (it d oesn‟t have to be)? Is the sou rce an SN MP agent know n to N N M by SN MP OID in oid_to_type (see p age 7)? Is the MIB for the trap ‟s enterp rise load ed ? Brow se the MIB u sing the MIB Brow ser or u se the snmpwalk com m and . Try to control SN MP trap s from the agents d irectly first. Every d evice m anu factu rer p rovid es a d ifferent m ethod ology for configu ring the SN MP agent and enabling and d isabling SN MP trap s (no stand ard configu ration interface or conventions exist for SN MP agent ad m inistration or cu stom ization ). Typ ically, these p aram eters are set throu gh the d evice‟s IOS. Lots of p op u lar SN MP MIBs can be fou n d in th e $OV_SN MP_MIBS d irectory, and som e of these hold trap m acros. To d eterm ine if a p articu lar MIB hold s trap m acros, look (grep ) for the keyw ord “N OTIFICATION -TYPE” insid e the MIB file. If that fails, tr y the vend or‟s w eb site, a general w eb search or a p u blic MIB rep ository. Page 141 lists several good MIB rep ositories. All p rivate enterp rise SN MP agent‟s OIDs are of the form : .1.3.6.1.4.1.number... N umber is the IAN A-assigned vend or ID. Any d otted nu m bers beyond the nu m ber im m ed iately follow ing .1.3.6.1.4.1 are vend or -sp ecific d esignations. The m aster list of enterp rises‟ nu m ber assignm ents can be fou nd at: w w w.iana.org/ assignm ents/ enterprise-num bers N ote that the vend or ID m ay often be m islead ing d u e to the m ergers and acqu isitions the high tech ind u stry is su bject to. Cisco is 9, H P is 11, Bay N etw orks is 18, etc. 109 Fognet’s Field Guide to OpenView NNM If there are lots of Cicso OID trap s com ing throu gh w ith no form at, i.e., the OID starts w ith 1.3.6.1.4.1.9, then see the p roced u re for form atting trap s on p age 108. It m ay be easiest to log into cisco.com 's CCO, and search for the N N M Integration Utility and install it. This m ay not only solve this p roblem , bu t m ight also p rovid e cu stom sym bols for Cisco d evices as w ell. Finally, it m ay be d esirable to not form at u nform atted trap s. One reason m ay be that it‟s too tim e consu m ing to track d ow n or bu ild the trap d efinitions. Another reason m ay be that the u nform atted trap s convey enou gh inform ation in their varbind s that a form atted trap d efinition isn‟t necessary. In highly-scaled environm ents, trap d efinitions consu m e m em ory and CPU and it m ay even becom e d esirable to d elete existing trap d efinitions, forcing them into the enterp rise d efau lt. In either case, if “no form at” events are frequ ent and exp ected , change the event m essage to sim p le p ass the event d ata (for exam p le, $*) so the events d on‟t convey that there is som ething w rong w ith N N M‟s configu ration. Agent in distress; “spinning in ifTable or ipAddrTable” This hap p ens w hen an SN MP agent rep orts bad or corru p t inform ation. Generally the agent vend or m ay need to be contacted for ad vice. “Sp inning in ifTable” m eans that the SN MP agent has rep orted few er interfaces p resent in the variable ifN u m ber than p resent in the rou ter tables. This is a know n p roblem w ith som e Cisco PIX rou ters that w as fixed by new er cod e. If this is hap p ening, it cou ld cau se p erform ance p roblem s on the d evice the agent is ru nning on and shou ld be im m ed iately ad d ressed . In other w ord s, the agent really is in d istress and m ay be ru nning loop s that consu m e CPU and / or m em ory. “Sp inning in ip Ad d rTable” m eans that the SN MP agent is rep orting the sam e IP Ad d ress for the sam e interface (ifInd ex) m ore than once in the table. There m ay be other reasons these alerts are being generated , bu t alm ost alw ays, it is becau se there is a p roblem o n the target agent, not w ith N N M. OV_PhysAddr_Mismatch This event is generated by H P Op enView . One com m on reason to see this event is d u e to p roxy-ARP from the rou ters and / or th e d efau lt 110 Interpreting Events rou ting config on end -nod es, w hich N N M has trou ble w ith. ProxyARP is an alternative to d efining large rou ting tables on ed ge d evices, and m ore can be learned abou t it by read ing RFC 1027. In the case of a host w ith a d efau lt rou te, its ARP cache has the rou ter's MAC ad d ress as the MAC ad d ress for every non -local host it has com m u nicated w ith. The ARP-cache of the rou ter to w hich the non -local hosts are connected w ill have the ap p rop riate real entries, as w ill the actu al hosts. When OV sees the ap p arent d iscrep ancy, it generates this event. If Proxy-ARP is in u se, it is com p letely reasonable to change this event from an error alarm to log-only or to IGN ORE. The sam e can be said for the OV_ARPChgN ew PhysAd d r event. Inconsistent subnet mask Most com m only, the d evice‟s configu red m ask is actu ally incorrect. Som etim es, the first d evice d iscovered in a new netw ork is an im p rop erly-configu red netm ask, so su bsequ ently-d iscovered d evices w ith p rop er netm asks generate a flood of these events. This event is also generated becau se N N M ap p lies a classfu l m ask becau se it d oesn‟t have enou gh inform ation to p rop erly d eterm ine the correct m ask. In this case, N N M is the cu lp rit. For exam p le, su p p ose the first d evice d iscovered in the entire N N M top ology that has a 10 ad d ress is 10.1.142.5. netmon asks the d evice for its m ask w ith ICMP m ask requ est, bu t p erhap s the d evice d oesn't su p p ort that or filters that qu ery ou t. netmon then qu eries the d evice via SN MP bu t the d evice d oesn't answ er that either. In this case, N N M creates a netw ork object called 10.0.0.0/ 8, based on the classfu lness of the 10 netw ork, and p laces the new nod e in this new netw ork in the top ology. netmon su bsequ ently d iscovers 10.1.142.10 and asks it for the netw ork m ask w hich answ ers 255.255.255.0, w hich N N M then calls inconsistent becau se it believes it shou ld be 255.0.0.0. If a nam e for the corresp ond ing su bnet nu m ber isn't fou nd in / etc/ netw orks, then N N M nam es the netw ork sym bol as the su bnet nu m ber. N N M rem oves trailing zeros from the nam e. Therefore, 11/ 8, 11.0/ 16, 11.0.0/ 24 and 11.0.0.0/ 30 m ight all look like "11" w hen nam ed p er this m ethod . If nod es exist in both the 11.0 and 11.0.0 netw orks, inconsistent su bnet m ask events w ou ld then be generated for the nod es in the netw ork that w ere d iscovered later; after the m ask w as set for the “11” netw ork. 111 Fognet’s Field Guide to OpenView NNM The solu tion is sim p ly to ad d / etc/ netm ask entries, d elete, than red iscover the nod es. The netm asks file on Wind ow s resid es in the OS d irectory u nd er system 32\ d rivers\ etc. Finally, loadhosts m ay be u sed to force the m asks for netw orks to be created w hen they d on‟t yet exist in the top ology. DN S can also be u sed to configu re netw orks w ith corresp ond ing netnam es and netm asks, w hich gives better control over the netm asks. Another com m on reason this event is generated is w hen u sing variable-length su bnet m asks (VLSM), w hich N N M norm ally hand les fine. N N M su p p ort for VLSM w as ad d ed p rior to N N M 6.0. As d iscu ssed above, it's p ossible to have m ore than one su bnet m ask for a class A netw ork. For exam p le, if one u sed a 16 bit m ask (e.g., 10.1/ 16) and another u sed a 24 bit m ask (e.g., 10.2.1/ 24). This is OK - how ever, it w ou ld be "illegal" or w rong to have an interface w ith a 24 bit m ask that fell in the 10.1.x.x p attern, becau se it w ou ld be inconsistent accord ing to N N M. To fix this, valid ate the correct m ask for the netw ork and check if it is consistent w ith w hat N N M rep orts by right-clicking on the netw ork icon and d isp laying the object p rop erties; the IP Map attribu tes show the netm ask that N N M has assigned u p on d iscovery. If this is incorrect, ovtopofix –r 10 can som etim es fix the p roblem . If not, shu td ow n netmon u sing ovstop netmon, d elete the netw ork sym bol from all su bm ap s, and from all d efined m ap s, then red iscover the core d evice in the netw ork u sing loadhosts w ith the –m <m ask> op tion (p age 42). Restart netmon, confirm the p rop er netm ask is assigned to the netw ork sym bol, and issu e a d em and p oll to the d evice. In extranet environm en ts, the sam e sort of thing can hap p en to CIDRbased su bnets. If rnetstat -I on the offend ing d evice retu rns d ifferent netm asks for itself and its rou te, the netw ork settings cou ld be CIDR com p liant bu t not RFC 950 com p liant. In general, N N M is very strict on RFC 950 com p liance and if there are nod es w ith interfaces that have su bnet m asks "less" than th e class of netw ork in qu estion , then N N M w ill generate inconsistent su bnet m ask events. For exam p le, rnetstat –r <nod e> ou tp u t show s: 112 Interpreting Events Interface IP address Network Mask Network Address Loopback 10.177.104.9 255.255.255.252 10.177.104.8 BRI0 10.177.104.154 255.255.255.248 10.177.104.152 To fix it, d elete the incorrect interface from N N M and re-ad d it u sing the p rop er su bnet m ask. Use loadhosts (see p age 45) or m anu ally ad d the interface u sing m ap op erations. Rem em ber that CIDR is Classless Inter-Dom ain Routing, not classless A ddressing. RFC 1518 says that it sp ecifically d oes not ad d ress "p roced u res for assigning host IP ad d resses." It is a m istake to take rou ting p olicies and concep ts and try to ap p ly them to interfaces. If there is not an entry in the netm asks file or if the netm ask is not sp ecified w hen nod es are ad d ed u sing loadhosts, N N M u ses the d efau lt netm ask for the class of netw ork, and inconsistent su bnet m ask m essages m ay resu lt. OV_D uplicate_IP_Address There are several reasons these m ay be generated other than a m isconfigu ration in the m anaged environm ent. The m ost com m on reason is the p resense of H SRP or VRRP d evices w hich have m igrating IP ad d resses. The m ost recent versions of N N M d etect H SRP interfaces and treat them ap p rop riately via ET and the APA, bu t netmon m ay still com p lain. To stop this com p laining, these interfaces can be ad d ed to the netm on.noDiscover file and then d eleted from the top ology . In m any cases, backu p d u p licate interfaces are flagged as ad m inistratively d ow n in the MIB tables, so som etim es m anaging these via SN MP m ay w ork if p olling is via netmon. If d esired , the OV_Du p licate_IP_Ad d ress event can be su p p ressed for this class of d evice u sing the “-I” LRF sw itch for netmon (see p age 5 for u p d ating LRFs). When an IP ad d ress is m oved from one d evice t o another, N N M m ay generate OV_Du p licate_IP_Ad d ress events if the m ove w asn‟t “com p lete.” In m ost cases, d eleting the objects and red iscovering them solves the p roblem u sing the p roced u re on p age 195. It m ay not solve the p roblem , thou gh, if for exam p le DN S w asn‟t p rop erly u p d ated , or if the interface w as set to ad m inistratively d ow n instead of being rem oved from the sou rce d evice. OV_D uplicateIfAlias Du p licate ifAlias m essages m ay ap p ear after an u p grad e to N N M 6.31 or above. N N M u ses ifAlias by d efau lt for nam ing interfaces becau se they are a m ore stable w ay of tracking interfaces than u sing ifN am e 113 Fognet’s Field Guide to OpenView NNM (becau se ifN am e m ay change d u e to ind ex rem ap p ing by the SN MP Agent). Som e agents, how ever, au tom atically set the ifAlias to “testing” if the ifAd m inStatu s is set to “testing.” To elim inate these m essages, configu re the offend ing interfaces w ith d ifferent ifAliases if it‟s not too m u ch trou ble. Som e d evices‟ SN MP agents, how ever, m ay not p erm it a change in the ifAlias. To force N N M to revert to the legacy behavior of u sing ifN am e, set the follow ing netmon LRF sw itch u sing the p roced u re on p age 5: -k useIfAlias=false Another ap p roach is to m od ify the OV_Du p licateAlias event and set its logging behavior to “Ignore.” If u sing the APA p oller, the p aConfig.xm l file p rovid es m u ch m ore sop histicated m ethod s for hand ling ifAliases. Observe that som e p hysical interfaces show u p in the interface tables as several interfaces, and on the d evice it is only p ossible to configu re the SN MP ifDescr on one of them . In this case, the OV_Du p licatIfAlias alarm m ay be generated since the ifAlias m ay be id entical on the p rim ary interface and su b-interfaces. This behavior is com m on w ith ISDN BRI interfaces on Cisco d evices. You m ay w ant to u se netm on.interfaceN oDiscover for su ch su binterfaces. Authentication failures These generic SN MP trap s are generated by SN MP agents in resp onse to a failu re to p rop erly au thenticate an incom ing SN MP requ est. For exam p le, an SN MP get requ est w ith the im p rop er com m u nity string. Su ch a requ est cou ld very likely be generated by agents that are being contacted by the N N M server itself, since in general, it issu es a lot of SN MP requ ests. Therefore, these trap s typ ically ind icate the need to set the p rop er SN MP com m u nity string in N N M‟s SN MP Configu ration GUI. To d eterm ine the string N N M is u sin g to contact an SN MP agent, ru n: xnmsnmpconf -resolve <target> Most SN MP agents d o not p rovid e a facility to id entify the sou rce of the failed SN MP requ ests. Determ ining the sou rce of bad SN MP requ ests can be accom p lished u sing a sniffer (p age 172) to p u ll the 114 Interpreting Events sou rce from SN MP PDU of the offend ing requ ests. Detailed pmd tracing (p age 131) w ill d u m p SN MP PDU d ata as w ell. A few d evices d o p rovid e this d ata in p rop rietary trap s, how ever, and N N M‟s syslog facility m ay also be of u se in this case. Rem em ber, it ‟s som etim es hard to exp licitly d eterm ine the cau se of an Au thFail trap becau se it is a rather lim ited resp onse from an agent nod e to a nother nod e‟s attem p t to access its SN MP agent. To toggle SN MP au thentication failu res off at the sou rce, u se the SN MP agent vend or‟s im p lem entation-sp ecific m ethod s. Or, if the “w rite” com m u nity string is know n for the agent, u se: snmpset <node> snmp.snmpEnableAuthenTraps.0 integer 2 By d efau lt, the Au thentication Failu re event in N N M is set to “log only.” Consid er logging them as a secu rity or scalability p recau tion. At the very least, the event shou ld be set to log a statu s on occasion to valid ate that Au thFail trap s aren‟t cau sing p otential p erform ance issu es. Ru n ovdumpevents to view log-only events. It m ay not be a bad id ea to set u p a sched u led scrip t that grep s Au thFails fr om the event log and rep orts su m m ary find ings. In larger environm ents w here certain volu m es of Au thFail trap s are exp ected , set the logging behavior to “IGN ORE” to im p rove p erform ance of the N N M server . In the trap d escrip tion for the Au thFail trap , there is a note that claim s: “H P-UX only: On H P-UX agents, the valid com m unity nam es can be found in the / etc/ snm pd.conf file. The IP add ress that caused the authentication failure can be d eterm ined by either looking in the log file (snm pd .log), or by d oing a snm pwalk on the SN MP variable: .iso.org.dod.internet.private.enterprises.hp.nm .snm p. authfail.authFailTable.authFailEntry” N either the snm p d .log nor the stated MIB variable is ever u p d ated after bad requ ests to H P-UX instances of N N M. Still, the trap d escrip tion continu es to claim this to be so (erroneou sly). Unknow n or unrecognized ASN .1 type # messages These m ay show u p em bed d ed in SN MP trap varbind field s or in the p m d .log0 file and m ay cau se pmd and / or other d aem ons to crash. The cau se of these errors is u su ally m alform ed SN MP trap s. pmd stack tracing (see p age 131) m ay be requ ired to d eterm ine the trap sou rce. 115 Fognet’s Field Guide to OpenView NNM One solu tion is to d isable the transm ission of these p articu lar trap s from the sou rce and / or to seek u p d ated agent cod e from the vend or of the trap sou rce equ ip m ent. It m ay also be p ossible to form at a trap d efinition (see p age 95) that d oesn‟t u se the offend ing varbind s. Eliminating events from undesirable systems Regard less of w hether a nod e is in N N M top ology or not, SN MP trap s m ay be received from d evices that aren‟t “m anaged .” This is becau se there agents are configu red w ith the N N M server as a trap d estination . Often this can be reset at the agent. On system s that su p p ort p op u lar op erating system s, this is u su ally set in the snm p d .conf file. Stop p ing u nd esirable trap s at the sou rce is id eal. If ru nning V7.01 or above, the ovtrap d .conf file can be u sed to su p p ress trap s from p articu lar d evices. See p age 96 for m ore inform ation on ovtrap d .conf. If ru nning an old er version, there are several consid erations offered below . Som etim es a d evice is generating u nd esirable trap s for a sp ecific enterp rise. Within N N M, a trap d efinition for that p articu lar enterp rise can be d efined in the event configu ration and then configu red to “Ignore” all events. To force the N N M server to reject all trap s from all enterp rises for a p articu lar nod e, assign an incorrect com m u nity string to the target nod e in the SN MP Configu ration GUI. Sim ilarly, a n SN MP p roxy cou ld be configu red w ith a bad com m u nity string, for exam p le 127.0.0.1, so the N N M server no longer issu es requ ests to it. These tricks w ill cau se ad d itional tim eou ts and retries and m ay affect server p erform ance. Again, the m ost d esirable ap p roach is to ad d ress each offend ing nod e ind ivid u ally by configu ring its agent not to p rod u ce the trap s in the first p lace. 116 10. Event Correlation The ECS ru ntim e engine w as ad d ed to N N M in version 6.0 to su p p ort the su p p ression of cascad e failu re events. ECS circu its w ere su p p lied by H P, and the Correlation Manager w as p rovid ed as a front-end to ad ju st basic settings and to tu rn correlations on or off. Until N N M 6.31, there w as no facility in N N M for enhancing ECS correlation logic other than p u rchasing the exp ensive ECS Designer p rod u ct. The Correlation Com p oser , how ever, w as ad d ed in N N M 6.31 and it p rovid es an interface for creating ad vanced logic sets that are analogou s to ECS circu its, bu t it d oes not p rovid e the fu ll circu it d evelop m ent cap abilities of the Designer. The Com p oser is actu ally a front-end interface to a “su p er-circu it” in term s of ECS ru ntim e and the ECS Designer. The com p oser is actu ally im p lem ented as a regu lar ECS circu it. What is event correlation? Event correlation is u sed as a generic term to d escribe the ability to p reem p tively act on a stream of events in ord er to ad d intelligence to and control over that stream . One im p lication of event correlation in this context is that p ast events are related to cu rrent events in ord er to red u ce the overall am ou nt of events generated . To accom p lish this, event correlation tools m u st be “state aw are,” that is, they keep track of events over tim e. Many m od ern event correlation p rod u cts d ep loy finite d ifference engines internally to m aintain state, and a set of logical constru ct ions can be m anip u lated to form sp ecific logic sets (circu its) that then act on “stream s” of m essages, interru p ting or re-d irecting their flow throu gh the event m anagem ent system at variou s stages w ith great efficiency and scalability. When d iscu ssing event correlation in term s of N N M, it refers to a set of su bsystem s w hich are heavily integrated into the N N M p rod u ct. These 117 Fognet’s Field Guide to OpenView NNM su bsystem s inclu d e the ECS ru ntim e engine, the correlation m anager, the correlation com p oser, variou s configu ration entry p oints w ithin the N N M GUI and variou s configu ration files. Correlation Manager This is the legacy interface to event correlations bu ilt in to N N M starting w ith version 6.01. It p rovid es a JAVA-based GUI for changing p aram eters associated w ith N N M‟s bu ilt-in ECS circu its. Access the Correlation Manager throu gh the N N M m enu s: Options -> Event Configuration -> Ed it -> Event Correlation Or, the m anager can be invoked by URL: http :/ / host.fqd n/ OvDocs/ C/ ecs/ ecscm g.htm l (Window s) http:/ / host.fqd n:3443/ OvDocs/ C/ ecs/ ecscm g.htm l (UNIX) On UN IX installations, the p ort nu m ber (:3443) m u st be ap p end ed to the N N M Server hostnam e in the URL above. Correlation Composer The Correlation Com p oser is accessed via the Correlation M anager. The Com p oser can be lau nched by selecting “m od ify” after highlighting the “Com p oser” circu it in the Correlation Manager. Alternatively, in UN IX, the Correlation Com p oser can be started w ith the ovcomposer com m and . In Wind ow s, the com p oser can be started w ith: ovcomposer –m o ovcomposer –m d operator mod e (read -only) d eveloper mod e The Com p oser is a front-end interface to a “su p er circu it,” w hich is to say that it is a single ECS circu it that can be u sed to create a fairly w id e range of correlation logics sp ecifically for N N M. It p rovid es a w ay to ad d correlations for sp ecific SN MP events, and is bu nd led w ith N N M 6.31 and higher. In essence, the Com p oser is a fancy interface to create and m anip u late very fancy external ECS fact stores. The Com p oser p rovid es a few correlations that serve as exam p les as to the sorts of logic that can be created w ith this GUI. 118 Event Correlation The p ath to the correlator store is $OV_CON F/ ecs/ CIB and the N N M p rovid ed correlators w ith V7.5 inclu d e: N N MBasic OV_Connector_Interm ittentStatus OV_Chassis_Cisco OV_Multiple_Reboots N odeIf OV_N od eIf_Nod eDow n OV_N od eIf_Prim aryIFUnknow n OV_N od eIf_Nod eN otCorrelator Poller OV_Connector_Interm ittentStatus PollerPlus OV_Link_Interm ittent OV_Conn_Interm ittentStatus OV_Ad d r_Interm ittentStatus OV_Interface_Interm ittentStatus OV_N od e_Interm ittentStatus For creating cu stom correlations, the Com p oser p rovid es the follow ing correlator tem p lates: Enhance: Trigger the creation of a new output alarm Mod ify the inform ation for an existing alarm Multi-Source: Discard , mod ify, consolidate or create new output alarm (s) based on a set of m ultiple input alarm s Rate: Count like input alarm s and em it all or none plus a new alarm ind icating the threshold was breached Repeated : Mod e 1: Discard d uplicates in w ind ow of first alarm Mod e 2: Replace/ embed latest alarm in brow ser Sam e logic as DeDup correlation w ith more control Suppress: Discard alarm s matching cond itions Transient: Monitor rate and/ or consolid ate status-oriented alarm s Sam e logic as PairWise correlation w ith more control 119 Fognet’s Field Guide to OpenView NNM User-Defined : Alarm s m eeting ad vanced filter or sim ple alarm signatures invoke an input function such as a C or Perl script. ECS D esigner The Designer p rovid es a fu ll and robu st interface for ECS circu it d evelop m ent. The correlations p rovid ed w ith N N M w ere them selves d evelop ed w ith this tool (excep t for the Com p oser). With the introd u ction of the Com p oser, the ECS Designer is longer need ed to d evelop m ost correlations that m irror the form and fu nction of the existing correlations. For m ore ad vanced correlations, how ever, the ECS Designer is a m u st. The Designer p rovid es the interface for com p iling ECS raw circu it files (.ecs files) into com p iled circu its (.eco files) that can then be load ed into the ECS ru ntim e engine. The ECS Designer is not requ ired to load com p iled ECS circu its (.eco files) into the ru ntim e engine. This im p lies that circu its d evelop ed by som eone else can be shared (or, as is som etim es the case , sold ) to be load ed into an environm ent w here the ECS Designer is not installed . Instances and streams Insid e the ECS ru ntim e engine, events are sep arated into stream s, or flow s of events. All of the d efau lt ECS circu its and Correlation Com p oser instances are configu red to ru n on the d efau lt stream . It is very u nu su al to u se m u ltip le stream s in m ost environm ents. Even w hen events are com ing from d ifferent sou rces, for exam p le via pmd throu gh N N M, and via opcmsg throu gh OVO, the events all flow throu gh the d efau lt stream . Only in this w ay can events from d ifferent sou rces su ch as this be related to each other. Only w hen events com e from sou rces that m u st be com p letely sep arated shou ld sep arate stream s be em p loyed . An ECS instance is like a cop y of the ru ntim e ECS engine that is d ed icated to a p articu lar set of event sou rces. Mu ltip le instances of ECS ru ntim e can and d o ru n on single server and each are sep arate in their inp u t and ou tp u t. Circu its op erating in sep arate instances d o not interact w ith each other even thou gh they m ay all be configu red to u se the sam e stream at the sam e tim e. 120 Event Correlation The d efau lt instance is “1” and it is u sed for N N M‟s em bed d ed ECS ru ntim e. It gets its feed from pmd. ecsmgr com m and s are d irected to the d efau lt instance u nless the sp ecific instance is sp ecified w ith the “–i” op tion. So, for exam p le, the follow ing tw o com m and s are equ ivalent: escmgr –i 1 –info escmgr –info Tw o other instances that m ay be of interest are instance 11 and instance 12. Instance 11 is reserved for the OVO m anagem ent server and instance 12 is reserved for the OVO agent. A UN IX OVO m anagem ent server, then, m ay have three op erative instances of ECS ru ntim e, bu t they w on‟t be op erative by d efau lt, since there are no ou t of-box correlations for OVO server or OVO agent. Special correlations The DeDu p lication correlator (DeDu p ) is sp ecial becau se it is a p ostp rocessing correlator. This is requ ired in ord er to interact w ith the alarm brow ser in su ch a w ay as to “m ove” existing alarm s from their tim e-based p osition in brow ser. To accom p lish this, the correlation externally com m u nicates d irectly w ith the ovalarmsrv p rocess. The only configu ration entry p oint for this circu it is: $OV_CON F/ d ed up.conf Connector dow n event correlation circuit This correlation is m anaged by the Correlation Manager. N N M‟s first correlator, this ECS logic w as introd u ced in V6.0. Connector Dow n affects both netmon & APA statu s. Both netmon and APA internally d istingu ish betw een p rim ary and second ary failu res, and p ass this d ata to the correlation by w ay of varbind s. What this circu it d oes, how ever, is very d ifferent at a fu nctional level w ith resp ect to netmon versu s APA behavior. For netmon-based events, the circu it p erform s three layers of correlation w hich can be seen in the event brow ser as an event w ith em bed d ed events w hich them selves have em bed d ed events (see Figu re 10-1). At the top layer are nod e events w ith interface -level events em bed d ed below . For each interface event, second ary interface alarm s are su bsequ ently em bed d ed , rep resenting cascad e failu res beyond the p rim ary interface 121 Fognet’s Field Guide to OpenView NNM Figure 10-1 in term s of netw ork p ath. netmon bu ild s and m aintains p ath d ata on d aem on start for every interface in the top ology. Only w ith later versions d oes netmon issu e d ynam ic p olls based on connector statu s. For all versions after V6.0, thou gh, netmon d ou bles (by d efau lt) sched u led p olls to second ary interfaces. For APA-based events, all second ary alarm s are su p p ressed , so the circu it is instead u sed to correlate the three layers of event based on Ad d ress events, Interface events, and N od e events (see Figu re 10-2). Like netmon-based correlation, the nod e level is the top layer, w ith interface events em bed d ed below and ad d ress level events em bed d ed below that. N ote that APA-based p rim ary/ second ary d eterm ination is p erform ed by ovet_pathengine, not ovet_poll. In ET Dynam ic view s, second ary statu s in the top ology is set to u nknow n and in IPMAP, the statu s is u nchanged . For netmon-based correlation, an Im p ortant N od e filter d efines nod es that w ill alw ays be consid ered p rim ary. This is d efined u sing N N M‟s stand ard filter d efinition langu age. The genannosrvr p rocess feed s im p ortant nod e filter d ata to ECS. For APA -based correlation, im p ortant nod es are configu red in the p aConfig.xm l file. Im p ortant nod es alw ays get a statu s change w hen a d evice in the p ath to them becom es u nreachable. 122 Event Correlation Figure 10-2 N odeIf event correlation This correlation is m anaged by the Correlation Com p oser, and affects netmon statu s only. It w as introd u ced in V6.31. Also called “Rou ter/ Sw itch H ealth,” this correlator su p p lem ents the Connector Dow n circu it. N od eIf su p p resses sim p le nod e d ow ns and is central to the m ajor shift introd u ced in V6.31 from nod e-centric statu s events to Interface-centric statu s events for netmon. N od eIf also correlates the d ynam ic netmon p olls for neighbor interfaces also introd u ced in V6.31 (see p age 55). N od eIf‟s effect is that OV_IF_Unknow n and OV_IF_Dow n statu s alarm s from interfaces w ithin rou ters or sw itches are su p p ressed and em bed d ed if a corresp ond ing nod e-level statu s alarm (OV_N od e_Unknow n; OV_N od e_Dow n) is received w ithin the sp ecified tim e w ind ow . N od eIf su p p resses interface statu s alarm s from d evices other than rou ters or sw itches so that these alarm s never ap p ear. The assumption is that end-node status alarms are not desirable. N od eIf su p p resses interface statu s alarm s from u nu sed (u nconnected ) p orts w ithin rou ters or sw itches. 123 Fognet’s Field Guide to OpenView NNM N od eIf ad d resses the follow ing failu re scenarios that ConnectorDow n d oesn't: If few er than all the interfaces on a router or sw itch fail, d ispla ys the interface status alarm s If all the interfaces on a router or sw itch fail, d isplay the OV_N od e_Dow n alarm w ith the interface alarm s nested und erneath If a router or sw itch failure is a second ary failure, then all second ary events are em bedd ed , unless they pass the Im portant Nod e filter Repeated event correlation circuit This correlation is m anaged by the Correlation M anager. The fu nctionality of this correlator, w hich shou ld be consid ered a legacy circu it, is su p ersed ed by introd u ction of DeDu p correlation in N N M 6.4. The rep eated event circu it em bed s su bsequ ent m atches u nd er original event in alarm brow ser, increm enting the correlated m ess age cou nt. Contrast this to DeDu p , w hich d eletes and em bed s original alarm , effectively “m oving” the original alarm to the head of alarm list. This correlator is u sefu l for su p p ressing inform ational m essages, and is em p loyed on the netmon-based N od e_u p event. The N od e_u p event occu rs w hen “all interfaces are norm al or u nknow n,” so in the case w here one interface goes d ow n and com es u p again, a N od e_u p event m ay be rep eated ly generated . This circu it is u sefu l for ind icating the cond ition w ithou t m oving the m essage in the alarm brow ser each tim e the cond ition rep eats, w hich is w hat the DeDu p correlator w ou ld d o. PairWise event correlation circuit This correlation is m an aged by the Correlation M anager and it ap p lied to m ost statu s events generated by either netmon or the APA. The behavior of PairWise is very d ep end ent on the version of N N M and su bstantially changes betw een V6.31, V6.41, and again in V7.01. The behavior of PairWise is also loosely cou p led to DeDu p correlation behavior. N N M V6.0, V6.1, V6.2: Status alarm s are acknowledged if the parent event received w ithin 10 m inutes (Paired Tim eWind ow ). Child events are released im m ed iately to brow ser and associated actions are im m ed iately launched . There is no 124 Event Correlation red uction in the num ber of alarm s, as the p arent events not em bed d ed (suppressed ). N N M V6.31, V6.41: Status alarm s are deleted if the parent event is received w ithin 10 m inutes (Paired Tim eWind ow ). Child events are held (not released im m ed iately), and should not be seen in the alarm brow ser at all. Actions are launched only if the Paired Tim eWind ow expires or the parent event is d etected . N o alarm s are seen at all if parent event is received w ithin the w ind ow and if the Paired Tim eWind ow expires, the child event is released w ith the original tim e stam p to the alarm brow ser. Subsequently, the p arent event is em bed d ed , and should be subject to bein g d eleted by the DeDup circuit (see below ). N N M V7.01+: Child events are released im m ed iately to the alarm brow ser and associated actions are im m ed iately launched . Status alarm s subsequently d eleted if parent received w ithin 10m . Parent event received after w ind ow is sent to alarm brow ser, and are not em bed d ed . D eD up event correlation This correlator is m anaged by the Correlation Com p oser and affects netmon & APA statu s. It w as introd u ced in V6.41+. It is an im p rovem ent on the Rep eated Event in that it u ses p ost-p rocessing logic that takes feed from event d atabase as op p osed to the event stream via pmd. DeDu p d eletes and em bed s the existing m atching alarm , “rep lacing” it w ith the latest alarm . DeDu p w as cou p led to PairWise circu it in N N M 7.0+. Any child event d efined in DeDu p and PairWise shou ld be d eleted by the receip t of the p arent alarm , even after PairWise Paired Tim eWind ow exp ires. The affected “d ow n” statu s alarm s are alw ays d eleted from the alarm brow ser w hen m atched w ith their corresp ond ing “u p ” alarm . This correlator is configu red via an external file, $OV_CON F/ d ed u p .conf. Uncom m ent the follow ing line to entirely d isable this correlation: DEDUPLICATION=OFF Intermittent Status event correlation This correlation is m anaged by the Correlation Com p oser, w as introd u ced in V6.31, and affects netmon statu s only. This correlation is also called “Rou ter/ Sw itch Interm ittent Statu s” or 125 Fognet’s Field Guide to OpenView NNM “OV_Connector_Interm ittentStatu s.” Interm ittent Statu s show s flap p ing interfaces w hose statu s w ou ld otherw ise be su p p ressed by the PairWise correlation‟s behavior. This logic only ap p lies to connector interfaces, and it generates a new event, OV_IF_Interm ittent (Op enView enterp rise 58982423) to ind icate the flap p ing cond ition. The new event is generated w hen the inp u t event occu rs RATE_COUN T tim es w ithin RATE_PERIOD, w here by d efau lt: RATE_COUN T = 4 in V6.31 RATE_COUN T = 5 in V6.4 RATE_COUN T = 4 in V7.0+ RATE_PERIOD = 30 m inutes OV poller plus event correlation This correlation is m anaged by the Correlation Com p oser, w as introd u ced in V6.41, and affects APA statu s only. This correlator is u nload ed by d efau lt. This correlator has the sam e fu nctionality as Interm ittentStatu s, bu t ap p lies to the LinkDow n event and to APA statu s events affected by PairWise correlation. The correlator com p rises five ind ivid u al logic sets for APA connection, interface, ad d ress, nod e events and LinkDow n trap s. The new event generated w hen a flap p ing cond ition is d etected is of the form : OV_APA_[IN TERFACE| N ODE| ADDR| CONN ]_Interm ittent This alarm is generated w hen like alarm s p rocessed by PairWise correlation rep eat RATE_COUN T tim es w ithin RATE_PERIOD, w here: RATE_COUN T: Default is 2 in V7.01 RATE_PERIOD Default is 30 m inutes To enable this correlator (d isabled by d efau lt), a d d the follow ing entry to the $OV_CON F/ ecs/ CIB/ N am eSp ace.conf file, then red ep loy the correlators w ithin the Correlation Com p oser w ind ow : OV_Poller_Plus=PollerPlus.fs Multiple reboots event correlation This correlation is m anaged by the Correlation Com p oser and is not tied to Op enView statu s events. This correlator creates a new alarm w hen 4 (by d efau lt) or m ore cold Start or w arm Start trap s are received w ithin 5 (by d efau lt) m inu tes from a sp ecific SN MP -agent/ IP-ad d ress p air. This correlator is yet another rep eat of the fu nctionally of Interm ittent Statu s (see above). Introd u ced in V6.31, this correlator 126 Event Correlation w as d esigned to d etect m u ltip le reboots w hich w ou ld otherw ise never been seen p er the PairWise 6.31 behavior (see above). Cisco chassis event correlation This correlation is m anaged by the Correlation Com p oser and it m onitors the follow ing CISCO-STACK-MIB trap s and generates one of three new Op enView enterp rise alarm s if the cond ition p ersists for 10 m inu tes: .1.3.6.1.4.1.9.5.5 .1.3.6.1.4.1.9.5.6 The Op enView trap s generated are: OV_Chassis_Tem perature (specific 8982424) OV_Chassis_FanFailure (specific 58982425) OV_Chassis_Pow erSupply (specific 58982426) The circu it requ ires that the Stack MIB is load ed on the m anagem ent server and that the trap s are enabled for each Cisco d evice (they are d isabled by d efau lt). See p age 187 for instru ctions on how to enable these trap s. Summary of event correlations affecting status Figu re 10-3 show s som e of the interactions betw een N N M and the event correlation engine. Events flow into the ovevent stack w ithin pmd. There, events are d irected on p arallel tracks to d aem on that register w ith pmd. The d aem ons register for raw , correlated , or all ou tp u t from the ovevent stack. ECS is one of the d aem ons that register for this feed , bu t it is sp ecial in that it is able to feed its ou tp u t back into the ovevent stack as inp u t. 127 Fognet’s Field Guide to OpenView NNM Figure 10-3 128 Event Correlation Useful ECS manager commands The follow ing com m and s d isp lay Engine Statu s and d etails: ecsmgr ecsmgr ecsmgr ecsmgr –info –stats –stats verbose –snapshot <output_file> The ecsmgr –stats com m and show s w hich ECS circu its are d oing w hat w ith each event they p rocess. If there is a cou nt associated w ith the u nd ecid ed stat, this gives the nu m ber of events that are cu rrently being held by the circu it in som e sort of tim e w ind ow . The ecsmgr –snapshot com m and d u m p s d etailed inform ation abou t the states of op erating circu its. It w ill also show d etails abou t the events that are cu rrently being held by each ECS circu it. To externally d isable/ enable ECS circu it: ecsmgr –disable PairWise ecsmgr –enable PairWise To externally change ECS Circu it, m od ify the .d s file, then: ecsmgr –disable PairWise ecsmgr –circuit_reload PairWise ecsmgr –enable PairWise Configuring ECS circuits When N N M-su p p lied ECS circu its are configu red u sing the Correlation Manager to change p aram eters su ch as tim e w ind ow s, etc., the follow ing files shou ld get u p d ated au tom atically. Som etim es, errors are p rod u ced after changing p aram eters. If so, check these files: $OV_CON F/ ecs/ circuits/ <circuit_nam e>.d s $OV_CON F/ ecs/ circuits/ C/ <circuit_nam e>.param D isable or enable ECS ECS ru ntim e and d efau lt ECS correlators are enabled by d efau lt w hen N N M is installed . To p erm anently d isable ECS, a d d the follow ing sw itch to p m d .lrf (p roced u re on p age 5): -SOV_EVENT;p0 Disable ECS on the ru nning p m d : pmdmgr -SOV_EVENT\;p0 To re-enable ECS after rem oving the -SOV_EVENT;p0 LRF flag, ru n: pmdmgr -SOV_EVENT\;p1 129 Fognet’s Field Guide to OpenView NNM To d isable all ECS circu its on the N N M instance u ntil the system reboots or the ovstart is issu ed , ru n: ecsmgr –reset To re-enable all circu its after a reset: For each .d s file in: $OV_CON F/ ECS/ circuits: ecsmgr –enable <circuit_name> Or, m anu ally toggle the correlations back on in the GUI. Logging incoming ECS events To cap tu re all events entering the ECS engine: Turn on logging Turn off logging: Change log size: ecsmgr –log_events input on ecsmgr –log_events input off ecsmgr –max_log_size event <Kbytes> The log file nam e is ecsin.evt0. It is a rolling log, and is 512k by d efau lt. It is located in: $OV_LOG/ ecs/ 1/ ecsin.evt0 and $OV_LOG/ ecs/ 1/ ecsin.evt1 Logging output and correlated events To cap tu re events (inclu d ing ECS-created events) ou tp u t or d iscard ed by the cu rrently enabled correlators: ecsmgr –log_events stream on ecsmgr –log_events stream off ecsmgr –max_log_size event <Kbytes> The log file is nam ed d efau lt_xxx.evt0. It is a rolling log, it is 512K by d efau lt, and is located in: $OV_LOG/ ecs/ 1/ d efault_sout.evt0 $OV_LOG/ ecs/ 1/ d efault_sout.evt1 Events w hich are d iscard ed by the stream (or su p p ressed by a circu it) are w ritten to: $OV_LOG/ ecs/ 1/ d efault_sd is.evt0 $OV_LOG/ ecs/ 1/ d efault_sd is.evt1 Simulate events for testing ECS logic ecsevgen and ecsevout in $OV_CON TRIB/ ECS can be u sed to rep lay event logs for testing the effects of ECS circu it changes. For m ore info, see Trou bleshootingEventCorrelation.txt in that d irectory. 130 Event Correlation ECS tracing ECS tracing can be extrem ely verbose, bu t certain d evelop m ent activities can‟t be su fficiently trou bleshot w ith ou t it. To enable fu ll ECS tracing: ecsmgr -i 1 -trace 65536 pmdmgr -D0xffffffff -SECSS\;T0xffffffff The traces are then w ritten to $OV_LOG / p m d .trc0. Stack tracing via pmdmgr To tu rn on the tracing of the OV_EVEN T stack: pmdmgr -SOV_EVENT\;T0xffffffff This shou ld p rod u ce entries in $OV_LOG/ p m d .trc0. To tu rn on pmd d ebu gging to trace all stacks (SN MP, etc): pmdmgr -D0xffffffff or pmdmgr -D0xffffffff -SECSS\;T0xffffffff To m ake su re pmd p roblem s are not related to ECS, before starting pmd, renam e or m ove the ECSS stack configu ration file: / etc/ opt/ OV/ share/ conf/ stacks/ pm d/ ECSS.cfg To tu rn tracing off, sim p ly restart OV, or ru n: pmdmgr -SOV_EVENT\;T0x0 Monitoring pmd bottlenecks (N N M 6.1 only) This version only show ed pmd p erform ance in ovstatu s ou tp u t. This rep orting w as later rem oved , as it w as a d evelop m ent d ebu gging sw itch that w as accid entally left on. Ru nning „ovstatus pmd‘ w ou ld rep ort: pmd RUNNING Statistics: cce=3, rqts=0, memory unfreed=6 "rqts" is nu m ber of p end ing op erations in the pmd event p rocessing qu eu e. A valu e above 50 cou ld m ean bottlenecks. Correlation Composer tracing Follow these step s to enable Com p oser tracing. 1. Enable tracing w ithin the Com poser GUI 131 Fognet’s Field Guide to OpenView NNM 2. 3. 4. 5. cd $OV_CONTRIB/ ecs/ CO Run: ecsmgr –fact_update Composer CompTraceOn.fs Run: pmdmgr –SECSS\;T0xffffffff Tracing output is w ritten to $OV_LOG/ pm d .trc0 To d isable Com p oser tracing: 1. Run: pmdmgr –SECSS\;T0x0 2. Run: ecsmgr –fact_update Composer CompTraceOff.fs 3. Disable tracing w ithin the Com poser GUI 132 11. SNMP Functions Up on installation, N N M im p lem ents a p rop rietary SN MP agent infrastru ctu re (Em anate) d esigned to accom m od ate m u ltip le SN MP agents on a single server (the N N M m anagem ent station). This allow s N N M‟s sp ecial agent to co-exist w ith OS-oriented agents, therefore allow ing for extensibility. The Em anate m aster and su bagents are OEM‟d from SN MP Research, Inc. as p art of the N N M p ackage. SN MP Research is also the p rovid er of the SN MP Secu rity Pack that is requ ired for N N M to su p p ort SN MPv3 agents. SN MPv3 p rovid es strong secu rity su p p ort, am ong other ad vanced SN MP featu res. As of N N M 7.5, the SN MP Secu rity Pack com es bu nd led w ith the AE version, bu t only the Em anate agent is installed by d efau lt. The Secu rity Pack inclu d es an ap p lication to rem otely configu re SN MPv3 u ser cred entials, access rights, and keys on the m anaged nod es. The SN MP Secu rity Pack requ ires licensing throu gh SN MP Research and it is available for all p latform s and versions su p p orted by N N M. N N M 7.5/ Secu rity Pack integration also p rovid es su p p ort for ET d iscovery via SN MPv3. Installation of the SN MP Secu rity Pack rep laces the Em anate agent w ith SN MP Research‟s m ore ad vanced Brass agent. SN MP versions and history SN MPv1 and V3 are fu ll IETF stand ard s. SN MPv2 never achieved stand ard statu s. There are three versions of non-stand ard SN MPv2 that have been im p lem ented by a variety of vend ors: V2C, V2U, and V2*. SN MPv2C is su p p orted natively in N N M. SN MPv1 ap p eared in 1988 u nd er RFC‟s 1065, 1066, and 1067. It is still w id ely u sed bu t has been criticized for its u se of client au thorization via cleartext com m u nity strings. SN MPv2, som etim es also referred to as SN MPv2P, w as initially introd u ced throu gh RFC‟s 1441 and 1452. It offered im p rovem ents of SN MPv1 and im p lem ented a p arty-based secu rity system w hich som e view ed as overly com p lex. SN MPv2C (as p rop osed in RFC‟s 1901 thou gh 1908) introd u ced a com m u nity-based secu rity m od el w hich com p rised SN MPv2 w ithou t the controversial 133 Fognet’s Field Guide to OpenView NNM new SN MPv2 secu rity m od el, u sing instead the fam iliar com m u nitybased secu rity schem e of SN MPv1. SN MPv2U (as p rop osed in RFC 1909) attem p ted to offer greater secu rity than SN MPv1, bu t w ithou t incu rring the high com p lexity of SN MPv2. A variant of this w as com m ercialized as SN MPv2* and the m echanism w as eventu ally ad op ted as one of tw o secu rity fram ew orks in SN MPv3. SN MPv3 as d efined in RFC‟s 3411 and 3418 achieved fu ll stand ard statu s in 2004. It encom p asses strong secu rity, p rovid es rem ote configu ration, has ad vanced scalability and d istribu tion featu res, and is com p atible w ith d ep loyed p reviou s SN MP versions. Configuring SN MP community strings on N N M agent Com m u nity strings for N N M‟s cu stom SN MP agent (Em anate) are configu red in the snm p d .conf file. On UN IX, this file is in : On Wind ow s , this file is in: / etc/ Snm pAgent.d %OV_CON F%\ SN MPAgent N ote that Em anate d oes not allow the w rite and read com m u nities to be set to the sam e string. Configuring SN MP versions N N M su p p orts SN MPv1 and SN MPv2C, and SN MPv3 w ith the SN MP Secu rity Pack. To sp ecify SN MPv2C as the p rotocol to u se for a p articu lar agent, u se xnmsnmpconf w ith the –setV op tion. To d eterm ine the version, u se: xnmsnmpconf –getV <target> Mu ltip le p rotocols can be sp ecified , for exam p le: xnmsnmpconf –setV <target> 1,2C Configuring multiple SN MP agents on a single node Som e d evices (e.g. servers) can su p p ort m u ltip le SN MP agents if those agents are bou nd to non-conflicting p orts. N N M su p p orts only one SN MP p ort p er IP H ostnam e, thou gh. If the target system had m u ltip le ad ap ters, or if virtu al IP ad d resses w ere assigned to an interface, sep arate SN MP agents cou ld w ork if each ad d ress w ere assigned a sep arate DN S nam e so that N N M cou ld d iscover them as thou gh they w ere tw o nod es. This is actu ally how SN MP agent sim u lators like Gam bit‟s MIMIC w ork. If the system is the N N M server, the Em anate extension agent can be u sed to ad d SN MP agents. On Wind ow s, 134 SNMP Functions wpaagt load s extension agents. The ad ap ter read s this registry key to find the extension agent DLLs to load : H KLM\ SYSTEM\ CurrentControlSet\ Services\ Paramters\ ExtensionAgents On UN IX, the extension agent is naaagt and is configu red via the file: / etc/ Snm pAgent.d/ naa.cnf. More info in can be fou nd in the agent‟s resp ective m an/ ref p ages. A creative w ay to su p p ort m u ltip le m onolithic agents: is to configu re som e extensible agent that is bou nd to p ort 161 to p roxy requ ests to the agents that are bou nd to other p orts. For m ore on extensible agents, see p age 146. SN MP master agent sw itches To enable tracing on UN IX p latform s, first stop the d aem on u sing: / sbin/SnmpAgtStart.d/K03SnmpMaster To enable tracing on Wind ow s p latform s, first stop the N T m aster agent Service. For both p latform s, once the agent has been stop p ed , ru n: snmpdm -apall -start The trace file shou ld be either / var/ ad m / snm p d .log or / tm p / snm p d .log. For SN MP Packet d u m p ing, see the –hexdump – vbdump op tions. To re-install, ru n: snmpdm –stop –remove snmpdm –install SN MPv3 configuration using SN MP security pack Once the SN MP Secu rity Pack is p rop erly installed , configu re the SN MPv3 agents throu gh the stand ard SN MP Configu ration GUI (xnmsnmpconf) and enter the au thentication d ata u sing the keyw ord as p art of the com m u nity string, e.g: "3A;AuthPassword/AuthUser" SN MPv3 offers three levels of secu rity w here Au thorization (Au th) equ ates to u ser au thentication and Privacy (Priv) equ ates to d ata encryp tion. The Secu rity Pack secu rity m od e p refixes (like that u sed in the exam p le above) corresp ond to the levels as follow s: 3N; noAuth/ noPriv: supply usernam e only, e.g. “guest” 135 Fognet’s Field Guide to OpenView NNM 3A; 3P; Auth/ noPriv: Auth/ Priv: supply pass phrase (user/ pass) supply pass phrase (user/ pass) SN MPv3 Security Pack installation Follow these step s to install the SN MPv3 Secu rity Pack: 1. Install the Pack (UNIX target is / opt/ Snm pri/ SecurityPack) 2. Make sure the process brassd is running: ovstatus –c brassd 3. Ed it the configuration files for the brassd daem on: / etc/ srconf/ agt for agent / etc/ srconf/ m gr for the manager sid e of SN MPv3 process 4. Configure the usernam e/ password for SN MPv3 5. Configure the d evices to allow SNMPv3 protocol N ote that the Pack rep laces the Em anate Agent w ith brassd. Log file: Comm and s d ir: MIB d irectory: / opt/ Snm pri/ SecurityPack/ log / opt/ Snm pri/ SecurityPack/ brass/ bin / opt/ Snm pri/ SecurityPack/ m ib Exam p les of SN MPv3 qu eries: snmpwalk -v 3 -l authNoPriv -a MD5 -u usr -A pwd <node> snmpwalk –c 3A;authPassword/authUser <target> Configuring and controlling trap destinations The N N M server attem p ts to set itself as the trap d estination for any d evice it d iscovers. Most SN MP agents are fairly qu iet w ith resp ect to the am ou nt of SN MP trap s they generate, bu t there are d efinitely excep tions. H P netw ork p rinters, for exam p le, can be very chatty by d efau lt. To globally p revent N N M from setting itself as an agent‟s trap d estination, u se the “-N ” LRF flag to netmon (see LRF Proced u re on p age 5). To p revent N N M from setting itself as an agent‟s trap d estination for a class of d evices, d isable SN MP d iscovery u sing the “ I” flag in the oid_to_type file (see p age 7). Prior to V6.0, netmon u sed (accord ing the netmon m an/ ref p age) a m echanism that “byp assed norm al SN MP au thentication m easu res.” After V6.0, netmon u ses the set com m u nity string p rovid ed in the SN MP configu ration (xnmsnmpconf) to attem p t to set the m anagem ent server‟s IP ad d ress as the trap d estination for any d iscovered SN MP agent. 136 SNMP Functions SN MPv3 and VLAN information There is a know n p roblem w ith N N M 7.53 and earlier versions of N N M w here if only SN MPv3 qu eries are m ad e to the d evice (via ET's hook into the BRASS agent), in m ost cases, the VLAN tables of su ch a d evice can not be qu eried d u e to the fact that the sp ecial character (@) that ET internally ap p end s to the configu red com m u nity string d oes not w ork for the sp ecial com m u nity string configu ration entries need ed for u se by the BRASS agent. Controlling the N N M server SN MP address Control over w hich interface on the N N M server is set for trap d estinations and ou tgoing SN MP requ ests can be changed u sing the $OV_CON F/ ov.conf file. See the ov.conf m an/ ref p age for d etails. Controlling an agent’s SN MP address The SN MP ad d ress u sed for an agent is d eterm ined d u ring netmon d iscovery u sing the algorithm listed on p age 26. By d efau lt, netmon tests other ad d resses if the d iscovered ad d ress fails to resp ond . This behavior is controlled by the –k pickSnmpAddrPolls op tion as d iscu ssed on p age 43. For APA p olling, the SN MP Ad d ress is initially d eterm ined by netmon d iscovery and is affected by –k pickSnmpAddrPolls as m entioned above, then fu ther refined throu gh p aConfig.xm l confiru ration settings. To force a nod e to u se an SN MP Ad d ress other than the one p icked by netmon d iscovery, see the p roced u re on p age 44.To control the p referred m anagem ent ad d ress op tions in the APA, see p age 92. Community string discovery This is a u sefu l featu re, bu t it is also a p ossible sou rce of p erform ance d egrad ation if it is u sed too generally in highly scaled installations. Configu re com m u nity string d iscovery u sing: $OV_CON F/ netm on.cm str Prior to V 7.53 Interm ed iate Patch 19, the hash character (#) w as an invalid character in com m u nity strings. After that p atchm that character is allow ed , bu t it m u st be contained w ithin d ou ble qu otes. Try to lim it the u se of w ild card s, p articu larly in highly -scaled environm ents. See p age 52 for inform ation abou t setting SN MP 137 Fognet’s Field Guide to OpenView NNM tim eou ts and retries. In the follow ing exam p le, p u blic1 is u sed for d evices in the 10.2 netw ork, and p u blic2 is tried if p u blic1 tim es ou t: "public1", "public2" : 10.2.*.* : : : CERT advisory CA-2002-03, SN MP vulnerabilities In 2002, OUSPG ran tests and fou nd issu es w ith SN MP requ est hand ling and w ith SN MP trap hand ling in both agents and m anagers. In essence, the ad visory said w hat everybod y everyw here all read y knew : that SN MPv1 is insecu re and its u se can exp ose system to exp loitation. It fu rther fou nd sp ecific vu lnerabilities for a lim ited set of SN MP agents that cou ld lead to DOS attacks, bu ffer u nd erru n exp loits, and other nastiness. The sou rce of these w as fou nd to be in the vend orsp ecific fu nctions w ritten to p arse ASN .1 form atted MIB d efinitions, bu t fortu nately not in ASN .1 itself. These vu lnerabilities w ere im m ed iately ad d ressed by m ost vend ors and by H P throu gh p atches to N N M 6.2 and other H P p rod u cts. More info on this at: w w w.cert.org/ ad visories/ CA-2002-03.htm l w w w.cert.org/ tech_tips/ snm p_faq.htm l SN MP PD U size limitations This is a concern w hen u sing d ata collections. When there are m any collections configu red , there m ay be excessive fragm entations attribu table to N N M SN MP op erations. RFC1157 states that "An im p lem entation of this [SN MP] p rotocol need not accep t m essages w hose length exceed s 484 octets. H ow ever, it is recom m end ed that im p lem entations su p p ort larger d atagram s w henever feasible." N N M has no su ch lim itations, bu t som e agents m ay ad here to this low er lim it. In p ractice, this is an excep tion and not the ru le. Accou nting for IP and UDP encap su lation, the theoretical m axim u m SN MP PDU size is 65467 bytes (m ax UDP d atagram size - m ax IP head er length - UDP head er length). It is very likely, how ever, that su ch an IP p acket is fragm ented , increasing the chance of losing the entire PDU, since m any MTUs for IP are set at 1500 bytes. In this case, the m ax PDU w ou ld be cu t into over 40 fragm ents, the loss of one of w hich d u e to a collision, for exam p le, m eans the loss of the entire PDU. For this reason, it is recom m end ed that SN MP PDUs be lim ited to ju st u nd er the m inim u m MTUs in the rou te betw een the server and target. In N N M, PDU size can be d ictated throu gh the “ -m ” LRF (see p age 5) op tion to snmpCollect and m onitored via the ovstatus –v snmpCollect com m and . Also, there w as a know n secu rity p roblem 138 SNMP Functions w ith invalid ly-form ed trap s w hose PDU sizes exceed ed 4.5k bytes w ith N N M version 6.2 and p rior. That sp ecific issu e w as ad d ressed via p atches. More on this CERT ad visory-related issu e can be fou nd below . N ote that SN MPv3 ad d ressed som e of the above-m entioned lim itations d irectly. ov t rapd daemon not starting (Window s) There is know n p roblem w ith N N M 6.x version on Wind ow s w h ich is cau sed by the fact that Wind ow s services start in rand om ord er, and ovtrapd is d ep end ent on WinSN MP service. The error that interm ittently occu rs on system boot u p is: ovtrapd is not running. Error message is: winsnmp snmpregister() failed: 100. exit 1 To resolve the p roblem , configu re ovtrapd to listen for trap s d irectly on p ort 162/ u d p and 0xc900f/ ip x and not register w ith the Wind ow s SN MP trap service. Do this by configu ring ovtrapd to u se the "-W" LRF sw itch (see p age 5). N N M w ith SMS or other SN MP tools (Window s) SMS (and several other tools w ith SN MP m anagem ent cap abilities) rep laces w snm p 32.d ll w ith their ow n versions or installs the Microsoft version of w snm p 32.d ll. When N N M is installed , it installs a cu stom version of w snm p 32.d ll. If there is a conflict, let the new er ap p lication u se its p referred version of w snm p 32.d ll and u se the –W LRF sw itch to ovtrap d .lrf. See p age 5 for LRF p roced u re. Binding SN MP trap reception to ports other than 162 ovtrapd can be m ad e to listen to other p orts for SN MP trap s. In the services file (/ etc/ services on UN IX, or on Wind ow s: ..\ w ind ow s32\ d rivers\ etc\ services), there is an SN MP trap entry set to p ort 162 (UDP), and this can be changed to a d ifferent UDP p ort. ovtrapd d aem on follow s the p ort bind ing of SN MP. pmd, how ever, u ses p ort 162 (TCP) for talking to other cop ies of N N M, so changing the SN MP trap receiving p ort to any other valu e can break the rem ote pmd com m u nications in environm ents u sing DIM or N N M to N N M event forw ard ing. SN MP manager command line utilities All the com m and s below are tied to N N M libraries, so they w ill only w ork on the N N M server: 139 Fognet’s Field Guide to OpenView NNM snmpget snmpwalk snmpset snmpbulk snmpnotify snmptrap xnmsnmpconf snmpcollect xnmbrowser xnmtrap xnmloadmib xnmevents xnmgraph mibtable rnetstat Query a single MIB variable Query an entire MIB table Set or change a w riteable MIB variable Query a MIB table via SN MPv2 GetBulk Issue an SN MP trap or inform request (d eprecated by snm pnotify) SN MP configuration utility Data collector background daem on MIB brow ser Event configuration utility MIB load er utility N ative event brow ser MIB graphing utility SN MP table data d isplay utility SN MP netw ork statistics utility N ote that m any SN MP m anager p ackages u se the sam e nam es for these p rogram s. For exam p le, Red H at Linu x AS 2.1 inclu d es the u cd snm p p ackage w hich inclu d es several u tilities nam ed the sam e. If ru nning N N M u nd er AS 2.1, the u cd -snm p u tilities w ith the sam e nam es as the N N M u tilities w ill ru n before the N N M su p p lied com m and s becau se / u sr/ bin is ap p ears in the p ath d efinition before / op t/ OV/ bin. Sim p le u p d ate the $PATH variable so / op t/ OV/ bin is before / u sr/ bin to get arou nd this issu e. It m ay be necessary to alo u p d ate the $MAN PATH variable in a sim ilar fashion. Strip OID and type data from SN MP query output This w orks on both UN IX and Wind ow s p latform s. N orm ally, SN MP qu eries retu rn a lot of garbage, e.g.: C:\>snmpget patchy ifEntry.ifDescr.2 interface interfaces.ifTable.ifEntry.ifDescr.2 : DISPLAY STRING- (ascii): 3Com 3C920 Integrated Fast Ethernet Controller (3C905C-TX Compatible) The follow ing com m and strip s the OID and d atatyp e d ata: C:\>snmpget patchy ifEntry.ifDescr.2 | %OV_BIN%\bin\perl\bin\perl -p -e "s /.*: //g;" 3Com 3C920 Integrated Fast Ethernet Controller (3C905CTX Compatible) Perl script: output a node’s IP address table via SN MP This is u sefu l for feed ing other tools or for trou bleshooting DN S issu es. It is also a good tem p late for u sing Perl and N N M‟s SN MP qu ery tools for bu ild ing rep orts, etc.: #! /usr/bin/perl $host = $ARGV[0]; $mib2 = '.iso.org.dod.internet.mgmt.mib-2'; 140 SNMP Functions $ipt = 'ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex'; @cmd = `/opt/OV/bin/snmpwalk $host $mib2.$ipt`; foreach $line (@cmd) { $line =~ /^$ipt\.(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/; $ip_addr = $1; print "$ip_addr\n"; } exit; Scrip t ou tp u t exam p le: peasoup # ./snmpIPtable.pl sunny 127.0.0.1 192.168.1.107 192.168.1.108 peasoup # SN MP w eb resources General SN MP Info: w w w.snm plink.org Many MIBS: w w w.oid view.com / m ibs/ d etail.htm l Many MIBS: w w w.m ibd epot.com Many MIBS: assure24.com/ databases/ snm p -mib/ private SN MP OID Registry: w w w.iana.org All OIDS: w w w.iana.org/ assignm ents/ enterprise-num bers OID translation: jaguar.ir.m iam i.ed u/ ~marcus/ snm ptrans.htm l Cisco specific info: w w w.cico.com/ warp/ public/ 477/ SN MP/ m ibcom pilers.htm l N N M d evice and protocol support: w ww .openview .hp.com/ prod ucts/ nnm et/ support/ d evice_support.htm l N N M d evice support: openview.hp.com/ prod ucts/ nnmet/ support/ d evice_requirem ents.htm l 141 12. Systems Management Several facilities exist w ithin N N M to help m anage system resou rces su ch as d isk sp ace and system p erform ance m etrics, bu t N N M is not a su bstitu te for a fu ll fu nction d istribu ted system s m anagem ent tool. Syslog integration facility Any system or d evice that u ses the syslog facility (RFC 3164, 3195) can p rovid e a feed to this light-w eight version of H P‟s Op enView Op erations (OVO) logfile encap su lator agent. This featu re w as integrated into AE versions of N N M starting w ith version 7.0 and it is only su p p orted on N N M ru nning on UN IX p latform s. If a syslog consolid ator is requ ired for Wind ow s environm ents, several third p arty p rod u cts are available. Kiw i is a p op u lar choice. The syslog integration featu re is covered in d etail in Section 6 (Section 7 for V7.53) of the Gu id e to Using Extend ed Top ology u ser m anu al. The V7.53 d ocu m entation is m ore extensive than p reviou s coverage. Also, help is available at the follow ing URL: http:/ / <nnm server>:3443/ OvCgi/ OvWebH elp.exe?Content=slgref If APA is enabled , certain event correlation logics w ill u se syslog events to increase the intelligence of the APA. Investigate w hat syslog events affect the APA by lau nching the correlation com p oser u sing the com m and ovcomposer –m o. Exam ine the OV_Poller* correlators. These correlations can be extend ed by ad d ing a d d itional syslog events to them in ord er to trigger APA p olls, etc. The agent converts syslog m essages to N N M events and u ses the sam e p rop rietary configu rable m essage p attern m atching exp ression anchoring langu age u sed in OVO. To activate the N N M syslog facility in “N N M Dep loym ent Mod e”, w hich m eans the syslog facility w ill w ork ind ep end ently of a p eer OVO installation, the follow ing generalized step s are requ ired : 142 Systems Management 1. 2. 3. 4. 5. 6. If running H P-UX 11.0, make sure DCE is installed . If running N IS, ad d opc_op user and opcgrp group on the NIS server. Otherw ise, ad d opc_op and opcgrp locally Use $OV_BIN/ovsyslogcfg to configure trap mappings Run: setupSyslog.ovpl -standalone -deploy Test For exam p le: Solaris: logger -p user.err %LINEPROTO-5-UPDOWN: Line protocol on IF test2, changed state to down H P-UX: logger %LINEPROTO-5-UPDOWN: Line protocol on Interface test2, changed state to down Use the logfile to trou bleshoot syslog configu ration or installation: $OV_PRIV_LOG/ setupSyslog.log To configu re a Cisco d evice to log to the N N M server: (set) logging server <nnmserver ip> (set) logging server enable (set) logging level 5 all Cisco logging levels: 0 1 2 3 4 5 6 7 emergencies alerts critical errors warnings notification informational debugging In som e cases, m essages m ay be logged to a d ifferent filenam e su ch as local7log instead of being logged to th e syslog or m essages file. If this is the case, m od ify syslog.conf to p oint local7 t o syslog, and the d o a kill –HUP <PID> on the syslogd PID. The follow ing file is w ritten to by the syslogTrap facility and this file can grow w ithou t bou nd s: / var/ opt/ OV/ tm p/ OpC/ m sgagtd f To lim it the size of that file and the other qu eu e files as w ell, enable Op C m essage qu eu e bu ffering by ad d ing the below entries to the op cinfo configu ration file in / op t/ OV/ bin/ Op C/ install: OPC_BUFLIMIT_ENABLE TRUE 143 Fognet’s Field Guide to OpenView NNM OPC_BUFLIMIT_SIZE 10000 OPC_BUFLIMIT_SEVERITY critical Managing systems w ith HOST RESOURCES MIB There are H OST-RESOURCES-com p liant (RFC 2790) agents that are available both com m ercially and for free. Som e su p p ort extensions that d o neat things like logfile scrap ing. Vend ors inclu d e H P, SN MP Research, and Concord (CA). w w w .net-snm p .org p rovid es a com p lete agent as w ell as SN MP m anagem ent tools for free. For H OST-RESOURCES MIB su p p ort on the N N M server itself, the p u rchase of SN MP Research‟s CIAgent is requ ired becau se the agent has to be a su bagent to the Em anate M aster agent for that one p articu lar host. An alternative is to configu re an SN MP agent that su p p orts H OST-RESOURCES to u se a p ort other than UDP 162. Be su re to sp ecify the d esignated p ort w hen m aking qu eries to this agent u sing the GUI or com m and line tools. Monitoring processes using HOST RESOURCES MIB Once an SN MP agent is in p lace that su p p orts H OST-RESOURCES, one can set u p a d ata collection on the N N M server. For each instance of hrSWRu nN am e that corresp ond s to the p rocess (as rep orted in the corresp ond ing instance of hrSWRu nStatu s), configu re the collection u sing the follow ing p aram eters: d on't store, check threshold s, and configu re a threshold event that rep orts if the valu e is > 1. The OID for hrSWRu nN am e is: .1.3.6.1.2.1.25.4.2.1.2. Monitoring disks using HOST RESOURCES MIB Once an SN MP agent is in p lace that su p p orts H OST-RESOURCES, on N N M, set u p a MIB Exp ression (see p age 149) p er below and then set u p a d ata collection (p age 147) u sing that exp ression. FreeDiskMB \ ―( hrStorageSize–hrStorageUsed) times \ hrStrorageAllocationUnits\ divided by 1024*1024 ― \ 1.3.6.1.2.1.25.2.3.1.5 \ 1.3.6.1.2.1.25.2.3.1.6 – \ 1.3.6.1.2.1.25.2.3.1.4 * 1048576 / Managing systems using RD MI (Window s) Wind ow s versions of N N M su p p ort RDMI m anagem ent throu gh the ovcapsd d aem on. RDMI is H P‟s p rop rietary extension to access DMI 144 Systems Management m anagem ent inform ation rem otely. Only som e of H P‟s hard w are lines su p p ort RDMI. When N N M d iscovers any d evice, it tests for RDMI cap ability and sets the isRDMISu p p orted cap ability flag. Use the search fu nction to list RDMI su p p orted d evices that have been d iscovered . The em bed d ed DMI brow ser can then be u sed on those boxes. DMI as com p ared to SN MP: - The DMI Client is roughly equivalent to an SN MP Manager - The DMI Service Provid er is equivalent to th e SN MP Agent - The DMI MIF is equivalent to the SN MP MIB - A DMI Event is equivalent to an SN MP Trap Forw arding Window s events to N N M as SN MP traps The Wind ow s com m and line u tility evntcmd (or its GUI-based equ ivalent evntwin) can be u sed to configu re events to be forw ard ed to N N M via SN MP. The trap s com e to N N M u nd er the enterp rise 1.3.6.1.4.1.311.1.13.1. Best bet is to m anu ally create a new enterp rise for these trap s rather than try to chase d ow n any MIBs that m ight exist (they d on‟t). Forw arding N N M events to Window s event log Use the follow ing Perl cod e snip p et as p art of an action callback scrip t that p asses N N M event d ata to a Wind ow s Event Log: Use Win32::EventLog; my $ELog; my %event=( 'EventID',100, 'EventType',EVENTLOG_WARNING_TYPE, 'Category',NULL, 'Strings','This is a String', 'Data','THIS IS DATA', ); $ELog = new Win32::EventLog( 'MyScript' )||die $!; $ELog->Report(\%event) || die $!; Managing systems using Solstice enterprise agents (SEA) Defau lt installations of Solaris d o not have SN MP agents enabled . SEA ship s free w ith m ost versions of SUN ‟s op erating system s and im p lem ents a SN MP m aster/ su bagent architectu re sim ilar to Em anate that su p p orts MIBII, DMI, and an SN MP-DMI cross m ap p er. For m ore inform ation, or to d ow nload see: w w w.sun.com / software/ entagents/ features.xm l 145 Fognet’s Field Guide to OpenView NNM Accessing WMI data via SN MP The third p arty SN MP Inform ant sharew are p rod u ct p rovid es a WMI to SN MP agent that installs p rop rietary MIB extensions on Wind ow s system s. It requ ires that the Microsoft SN MP agent is installed and that the SN MP Inform ant MIB is load ed on the N N M server. The basic agent w ith visibility to arou nd 60 m etrics is free, and extensions for m ore d etailed WMI objects are available throu gh the fu lly licen sed version. N N M server system management via SN MP (UN IX) The Em anate SN MP m aster agent and inclu d ed su bagents ship p ed w ith N N M for H P-UX and Solaris p rovid e H P p rop rietary system s m anagem ent extensions. Access to these can be fou nd via the ovw p erform ance m enu as “CPU Load ” and “Disk sp ace.” Extensible SN MP agents These agents p rovid e a p latform for d evelop ing cu stom SN MP objects and MIB trees. Som e of these agents p rovid e facilities and featu res that go beyond sim p le SN MP agent d evelop m ent. H P sells su ch an agent as d o other N MS vend ors. The N et-SN MP agent (w w w .net-snm p .org) is very p op u lar as w ell and also p rovid es SN MP get, set, and trap generation com m and s. SN MP Research‟s CIA Agent is also a p op u lar and com p rehensive solu tion, as is Concord ‟s System Ed ge solu tion. 146 13. Data Collection and Thresholds SN MP agents are d esigned to hold very sim p le d ata typ es. This allow s the agents to be very efficient, bu t m akes the job of the SN MP m anager very hard . Excep t in the case of RMON , SN MP agents d o not track trend s or keep history of d ata. Interface traffic, for exam p le, is held in the SN MP agents as cou nter valu es that sim p ly increm ent w hen traffic p asses throu gh. When the cou nter fills u p , it resets to zero. N N M‟s snmpCollect backgrou nd d aem on p olls MIB valu es, d eterm ines the d ifference betw een the cu rrent valu e and the last p olled valu e, d iscard s any cou nter w rap s or resets, then record s the d elta in the snmpCollect d atabase on the m anagem ent server. Interface u tilization , p erhap s the m ost com m only u sed N N M p erform ance m etric, is not available from d irect SN MP agents. To rep ort this, SN MP m anagers m u st p oll ifInOctets, ifOu tOctets, and ifSp eed , then ad d the cou nter valu es and m u ltip ly by 8 to get the total nu m ber of bits before d ivid ing by ifSp eed , and then m u ltip ly the resu lt by 100 to obtain a p ercentage, as follow s: (ifInOctets + ifOutOctets) × 8 ───────────────────── × 100 ifSpeed Math that com bines MIB variables is p erform ed in N N M by the MIB Exp ression featu re and is configu red in the m ibExp r.conf file. N ote that SN MP valu es for interface link sp eed m ay not be accu rate. Also, the form u la exp ressed above is fine for half d u p lex Ethernet interfaces, bu t others, like fram e relay and serial lines or fu ll d u p lex FastEthernet or GigabitEthernet interfaces carry fu ll d u p lex traffic. ifTyp e can tell them ap art, bu t snmpCollect isn‟t sm art enou gh to d istingu ish half d u p lex from fu ll d u p lex lines. Som e of this confu sion w as m itigated by the ad d ition of the ovexp rgu ru in V6.01. This w rap p er for MIB d ata and MIB exp ression d ata p erform s qu eries to d eterm ine w hat MIB variables are su p p orted before the d ata 147 Fognet’s Field Guide to OpenView NNM collector actu ally goes off and collects them , and it has som e lim ited abilities to d istingu ish collections by ifTyp e. xnmgraph is the foregrou nd p rocess that d isp lays SN MP d ata from history, real tim e feed s, or both. It is d esigned to read raw d ata from the snmpCollect d atabase. Data from snmpCollect is exp orted to the d ata w arehou se for trend rep orts. D ata collection best practices Review the d ata collections that are set by d efau lt w hen N N M is installed . Dep end ing on the version, som e collections m ay be very intensive for certain d evices. The N N M versions closer to V6.0 had the m ost liberal settings for these collections. In general, the d efau lt collections are in p lace to su p p ort the rep orting fu nction. Disabling SN MP d ata collections d irectly in the d ata collections and threshold s configu ration GUI can affect certain rep orts. Conversely, d isabling rep orts u sing the rep orter GUI d oesn‟t necessarily d isable the u nd erlying d ata collections. When setting threshold s, a nu m ber greater than tw o is recom m end ed as the valu e of “for X consecu tive sam p les.” Most SN MP-based d ata tend s to be “sp iky” by natu re and this gu arantees that only su stained threshold violations generate threshold alarm s. Do not set the p olling intervals too tight. Generally, 15 m inu te, 30 m inu te, and 1 hou r sam p ling intervals are best. While tighter p olling intervals tend to m ore accu rately gau ge actu al trend s by m ore accu rately cap tu ring d ata arou nd cou nter w rap s and better isolating d ata sp ikes, the best p ractice is to u se tighter intervals on only a sm all set of collections w hich are intend ed to be tem p orary w here the focu s is on sp ecific p roblem s. Too m any collections w ith tight intervals lead to p erform ance issu es that u ltim ately d egrad e all collections‟ integrity d u e to increased failed d ata collection p oints from tim eou ts, etc. If%util (or other % expression) is greater that 100% Typ ically, this is becau se the interface being m onitored is ru nning at fu ll d u p lex. The If%u til MIB exp ression is d esigned for half d u p lex lines only. The next m ost com m on cau se of this is that ifSp eed is set incorrectly at the agent. To verify the sp eed as rep orted by SN MP, select the nod e and select the Configu ration ->N etw ork Configu ration >Interface Prop erties m enu item in ovw. Another cau se m ay be that the 148 Data Collection and Thresholds rou ter is com p ressing d ata before send ing it over WAN links. In this last case, the SN MP agent is record ing the incom ing d ata before com p ression and/ or the ou tgoing d ata after d ecom p ression. Utilization m ay exceed 100% m ight be d u e to the p resence of 64-bit cou nters in the MIB. See section below on exhau sted 32-bit cou nters in the p resence of 64-bit cou nters. On fram e relay links, the ifSp eed is typ ically the line CIR, w hich is u su ally less than the m axim u m bu rst rate for the link, so it is not im p ossible to have u tilizations that exceed 100% on these typ es of interfaces. Yet another p ossibility is that inbou nd traffic and ou tbou nd traffic are not sam p led sim u ltaneou sly. This is d u e to nu m erou s factors affecting the snm p Collect p oller and the d ifference cou ld be several second s, p articu larly w hen line u tilizations ap p roach 100%. The u nd erlying MIB d ata is accu rate, bu t inaccu racy is introd u ced becau se the tim e intervals u sed to calcu late the d eltas are fixed , yet the actu al sam p led d ata d oesn‟t corresp ond exactly to those intervals. Typ ically, how ever, these tim ing issu es only accou nt for m inor d ifferences in the rep orted u tilization. Also, op ening u p the p olling interval red u ces the chance for this sort of error. Finally, som e p eop le are d eterm ined to som ehow com bine inbou nd and ou tbou nd u tilization on fu ll d u p lex links. While setting u p an exp ression that su m s the tw o rates and then d ivid es by tw o shou ld never exceed 100%, bu t this is a p oor p ractice and can hid e real p roblem s. For exam p le, if inbou nd is p egged at 100%, and ou tbou nd is averaging 20%, it w ou ld never be d etected since the exp re ssion w ou ld rep ort 60% “average” u tilization. The best p ractice is to sep arate the inbou nd and ou tbou nd collections on Fu ll Du p lex links. The best solu tion w hen d ealing w ith m ixed d u p lex/ half d u p lex interface environm ents to u se N N M‟s new er ovexprguru featu re, w hich is covered in m ore d etail below . MIB expressions using mibExpr.conf and mib.coerce As illu strated above, sim p le SN MP qu eries have to be com bined to m ake u sefu l grap hs. $OV_CON F/ m ibExp r.conf allow s the creation of MIB exp ressions. Math in these form u las u ses RPN (Postfix notation). See the m an/ ref for m ibExp r.conf 149 Fognet’s Field Guide to OpenView NNM One requ irem ent of creating MIB exp ressions is that MIB valu es u sed in those exp ressions m u st be of the sam e d ata typ e. If an exp ression seeks to com bine a gau ge typ e variable w ith an integer typ e variable, an error resu lts. mib.coerce is p rovid ed to coerce MIB variable to a sp ecific typ e so they can be u sed in exp ressions. MIB expression guru (ovexprguru) Introd u ced in N N M 6.01, ovexprguru is an N N M com m and that is u sed to p re-select the ap p rop riate MIB exp ression (mibExpr.conf) to u se to grap h certain generic collections based on the collected object‟s cap abilities. Its u se can be seen in the ARF files for N N M m enu bar p erform ance and fau lt grap hs. The com m and takes the follow ing generic keyw ord s and attem p ts to ap p ly the ap p rop riate m ibExp r from the m ibExp r.conf file: utilization packetrate averagepacketsize thruput errorrate percenterrors discardrate datarate percentdiscards framerate framedatarate framecongestion frameerrors frameeligiblediscards In the case of u tilization (and several others of the above a rgu m ents) ovexprguru p olls the ifTyp e table and com p ares the resu lts to a table of interface typ es that are hard -cod ed in the com m and . If the instance is a fu ll d u p lex interface then the in and ou t u tilization are grap hed sep arately. For a half d u p lex interface the in and ou t are com bined . For m ore info, see the ovexprguru m an/ ref. Defau lt Data collections based on MIB exp ression su ch as If%u til d o not u se the MIB exp ression gu ru by d efau lt. In ord er rep lace an existing collection for If%u til, follow these step s: 1. Ad d a new MIB Expression in the $OV_CON F/ MibExpr.conf file, e.g.: BetterIf%Util \ “Interface Utilization that applies the appropriate expression \ n\ Based on the ifType of the interface being monitored.”\ INDIRECT : utilization : COMBINED 2. In the Data Collections and Threshold s , select the If%Util Object and Ed it -> MIB Object -> Copy, then select the new expression from the list of expressions. 3. Suspend the If%Util collection and save. Run: ovstop snmpCollect ovstart snmpCollect 150 Data Collection and Thresholds N N M V7.51 Interm ed iate p atch 18 introd u ced an u nd ocu m ented op tion to enable it to d u m p SN MP p ackets. The op tion is “ -d .” Understanding and manipulating statistical thresholds Statistical threshold ing w as introd u ced in N N M V6.2. This featu re u ses stand ard d eviations on collected d ata that is exp orted to the Data Warehou se. Previou sly, only fixed threshold s w ere available. When configu ring a d ata collection object, “Fixed ”, “Both Statistical and Fixed ”, or “Either Statistical or Fixed ” can be selected u nd er Threshold Param eters. Stand ard d eviation calcu lations requ ire the MIB variables be stored in the snm p collect DB and exp orts of trend d a ta to d ataw arehou se are enabled . The d efau lt tim e intervals for statistics are as follow s: The d ata points are sorted into 3 buckets: Workhours (8am -5pm w eekd ays), WeekDayOffHours (5pm -8am w eekd ays), and Weekend s. These bu ckets can be changed u sing the follow ing configu ration file w hich d oes not exist by d efau lt: $OV_CON F/ statTimeRanges.conf There is a m an/ ref p age for this file w hich d escribes how to configu re it in d etail. The stats that are exp orted from the Data Warehou se to bu ild the stand ard d eviations are stored in a flat file: $OV_DB/ snmpCollect/snmpColStats.txt snmpCollect commands To d u m p configu ration d ata to $OV_LOG/ snm p Col.trace, ru n: snmpCollect -S To d elete snm p Collect d ata old er than 90 d ays (2160 hou rs): ovcoltosql -N -D 2160 To rem ove all collected d ata for the snmpCollect d atabase: rm –r $OV_DB/snmpCollect/ pingResponseTime, pingPercentRetry threshold events These threshold events com e from d efau lt d ata collect ions that ship w ith som e versions of N N M like 6.x (bu t not 7). They ind icate p otential 151 Fognet’s Field Guide to OpenView NNM p roblem s w ith netmon p olling or w ith SN MP d ata collections. They can be tu rned off or configu red from "Data Collections and Threshold s" from the "Op tions" m enu . These are actu ally "external" MIB exp ressions, and that m eans they are MIB exp ressions that are d efined externally to the m ibExp r.conf file, so are not exp osed to u sers. See the m an/ ref for m ibExp r.conf for m ore inform ation. D ata collection and device naming/IP address issues The snmpCollect d aem on stores d ata by IP ad d ress. When xnmgraph is invoked to read the d atabase, it p erform s a looku p to d eterm ine w hich IP ad d ress to u se to recall d ata. There are several cases w here xnmgraph fails to extract the ap p rop riate d ata from the snmpCollect d b. These exam p les inclu d e: Targets that use round -robin DN S Targets that use H SRP Cluster nod es Targets that have been renam ed or re-IP ad d ressed The d ata for the aged -ou t IP ad d resses can be retrieved by u sing the exp licit IP ad d ress. A good tip is to u se loop back for m anagem ent and rep orting for rou ters (see p age 26.) OpenView specifics reserved for custom events The stand ard threshold and rearm events are d esigned to be cop ied to create cu stom events that p rovid e m ore m eaning and facilitate cu stom actions. The follow ing sp ecific event id entifiers have been reserved for u se for this p u rp ose: 1001-9999 odd 1002-10000 even For threshold crossed or error events. Custom rearm or return to norm al events D ata collection and high speed links Most SN MP stand ard MIB cou nters are 32-bit. A DS3 line at fu ll band w id th cau ses a cou nter like ifInOctets to w rap in abou t 12 m inu tes. An FDDI line w rap s in less than 6 m inu tes. With a 64-bit cou nter, how ever, a 1 tbs (terabits p er second ) link w ou ld take nearly 5 years to w rap at fu ll band w id th. A cou nter w rap d u ring a p olling interval resu lts in an ignored valu e in collected d ata. One w ay to im p rove grap h accu racy in this case is to 152 Data Collection and Thresholds p oll high sp eed links m ore frequ ently to red u ce the nu m ber of ignored d ata p oints, bu t this can ver y qu ickly cau se ad verse p erform ance p roblem s (see p age 148). 64-bit cou nters w ere introd u ced along w ith SN MPv2 to resolve these issu es and thu s are only fou nd in agents that su p p ort SN MPv2 or greater. The 64-bit cou nters that corresp ond to the stand ard interface table MIB can be fou nd in the ifMib table, and echo their 32-bit cou nterp art‟s nam e save for “H C” for H igh Cap acity inserted into their nam es, so for exam p le: ifH CInOctets is fou nd at 1.3.6.1.2.1.31.1.1.1.6. See the su p p ort m atrix on p age 296 for inform ation on N N M‟s su p p ort for variou s versions of SN MP. Exhausted 32-bit counters w hen 64-bit is supported If the MIB su p p orts 64-bit cou nters, RFC 2233 (ifMib) m and ates su p p ort of the corresp ond ing 32-bit cou nters for backw ard com p atibility and states that these cou nters d o N OT reset betw een 32 64 2 - 1 and 2 - 1. H ere is the actu al verbiage: When 64-bit counters are in use, the 32-bit counters must still be available. They will report the low 32bits of the associated 64-bit count (e.g., ifInOctets will report the least significant 32 bits of ifHCInOctets). Most vend ors that su p p ly H C cou nters com p ly w ith this RFC. To test this, p oll both the 32-bit cou nters and the 64-bit cou nters (e.g. ifInOctets and ifH CInOctets, etc.) See if the 32-bit cou nters are increm enting w hen the valu es for the 64-bit cou nters are greater than 32 2 - 1. If the 32-bit cou nters are not increm enting that cou ld exp lain any u nd esirable resu lts. Mod ify the collections to u se the H C MIB variables. For those system s in the collection that d o not su p p ort the H C cou nters, sep arate them ou t and ap p ly the trad itional collections. N ote that ovexprguru is able to test for 64 vs. 32-bit cou nter su p p ort. Printing graphs (UN IX) The d efau lt ou tp u t form at for grap hs is xwd. To convert this ou tp u t into m ore fam iliar im age form ats, u se the conversion u tility at $OV_CON TRIB/ N N M/ Im ageMagik. To p rint to an attached or netw ork p rinter, for exam p le, a color LaserJet, com m ent ou t the d efau lt d ata for the .p rintcom m and listed in $APP_DEFS/ XN m grap h and rep lace w ith: *.printcommand: xpr -device pjetxl -rv -header "$OVTITLE" | lp -d <colourPrinterName> 153 Fognet’s Field Guide to OpenView NNM For m ore inform ation of m od ifying ap p -d efau lts settings, see p age 9. Insufficient memory w hen launching graphs (Solaris) This sp ecific error m essage relates to m axMallocPercent resou rce and occu rs m ostly on Solaris 2.7 servers. The error is cau sed by a too sm all of a stack size, w hich can been seen w ith the com m and : ULIMIT –a. To resolve, ru n: ULIMIT –s unlimited Exporting graphs to other image formats (UN IX) N N M grap hs can be saved by d efau lt in xw d form at. An excellent (and free) u tility to convert from xw d to a variety of p op u lar form ats is netp bm w hich can be fou nd at: http:/ / netpbm .sourceforge.net/ D ata collections on VLAN s SPAN or p ort m irroring can be u sed to m onitor VLAN traffic. Typ ically, a w hole VLAN is m irrored to a single p ort to be m onitored . Try not to oversu bscribe the d estination p ort, i.e., m irroring a 1000MB VLAN to a 100m b p ort. xnmgraph command examples All N N M‟s m enu -bar p erform ance, fau lt and configu ration grap hs are d irect invocations of the xnmgraph com m and line u tility. The com m and invocations can be seen in the $OV_REGISTRATION / C d irectory. For m ore info on ARF, see p age 6. Registration files for grap hs p rod u ced u sing the MIB Ap p lication Bu ild er are p laced in the ovm ib su bd irectory. Grap h serial interface u tilization u sing d u p lex MIB exp ressions (“2” is the ifInd ex of the target interface): xnmgraph -title "Duplex Interface Utilization" -mib IfInFDplxUtilization::2::::::,IfOutFDplxUtilization::2:: :::: <target> <target> is the ip ad d ress of the d evice and the '2' in betw een all of the colons is the ifInd ex of the interface to grap h. To grap h Cisco p ort bit rates, inclu d ing real tim e and collected d ata (+brow se): xnmgraph +browse –title ―Cisco Port Bit Rates‖ .1.3.6.1.4.9.2.1.1.1.6:label:3:::::: .1.3.6.1.4.9.2.2.1.1.8:label:3:::::: <target> 154 –mib Data Collection and Thresholds SN MP data collection terminology PD U. Packaged Data Unit. A term for a m essage of a given protocol com prising payload and protocol-specific control inform ation, typically contained in a head er. PDUs exist betw een the OSI layers of protocols, and are not specific to any particular layer. PDU is often used incorrectly to d escribe an SN MP Message. Packet. A d iscreet u nit of data sent across a packet sw itching network, e.g. IP, Fram e Relay, etc. Per RFC 1594: "‟Packet‟ is a generic term used to d escribe unit of data at all levels of the protocol stack, but it is m ost correctly used to d escribe application data units.” Packets contain data and a head er containing an ID num ber, source and d estination ad d resses, and error-control data and can be of fixed or variable size. Used interchangeably w ith datagram . D atagram. Per RFC 1594, "a self-contained , ind epend ent entity of d ata carrying sufficient inform ation to be routed from the source to the d estination com puter w ithout reliance on earlier exchanges betw een this source and d estination com puter and the transporting network." The d ifferentiator betw een a dat agram and a packet centers on this concept of self-sufficiency. Because of this, there is an im plication that d atagram s are UDP (connection -less) vs. TCP, w hich is connection -oriented , but this is not always true. MTU. The Maxim um Transm ission Unit is the largest packet that a given netw ork m ed ium can carry. Ethernet, for exam ple, has a fixed MTU of 1500 bytes, ATM has a fixed MTU of 48 bytes, and PPP has a negotiated MTU that is usually betw een 500 and 2000 bytes. Octet. Eight contiguous bits of data. A byte is often 8 bits of data, but not alw ays. An Octet is always 8 bits of data. Frame. In generic term s, a fram e is a basic logical unit in w hich bit-oriented data is transm itted . An Ethernet Fram e is equivalent to a packet. A Frame Relay fram e is a d ata unit containing a start-of-fram e (SOF) d elim iter, head er, payload , cyclic red und ancy check (CRC), and an end -of-fram e (EOF) d elim iter. The payload can be 02112 bytes, and the CRC is 4 bytes. Fragment. A piece of a packet, also called a runt. When a router is forw ard ing an IP packet to a network that has a m axim um packet size sm aller than the transm itted packet size, it is forced to break up that packet into m ultiple fragm ents. These fragm ents should be reassem bled by the IP layer at the d estination host. Payload. The Payload is the part that represents application inform ation and application overhead information in a set of d ata being processed or transported. System Insight (Compaq Insight) Manager MIBs The CIM/ SIM MIBs are load ed by installing the SIM/ N N M integration p ackage (URL for this p ackage is on p age 300). To m anu ally load the MIBs w hen not u sing the integration p ackage, load in the follow ing ord er: 1. cpqhost.m ib 5. cpqstd eq.m ib 155 Fognet’s Field Guide to OpenView NNM 2. cpqhealth.m ib 3. cpqsinfo.m ib 4. cpqsrvm n.m ib 6. cpqstsys.m ib 7. cpqthrsh.m ib 8. cpqups.m ib SN MP data collection vs. RMON RMON (RFC1757) and RMON -II (RFC2021) are u sed interchangeably below . The d ifference betw een SN MP and RMON is that the “M” in SN MP stand s for m anagem ent w hereas the “M” in RMON stand s for m onitoring. RMON d ata is aggregated by RMON p robes, or agents, and these are im p lem ented as SN MP MIBs. While RMON p rovid es a m u ch m ore efficient w ay to m onitor traffic at or nearer to the sou rce, for the m ost p art, vend ors are m oving aw ay from su p p orting RMON and relying m ore on SN MP and p acket cap tu ring tools to p rovid e the m onitoring fu nctions. N ote that N N M, how ever, d oes not p rovid e any p acket cap tu ring cap abilities ou t of the box. There are d ifferent "layers" of RMON , and d ifferent cap abilities that a nod e m ay im p lem ent. Most RMON im p lem entations d o not conform to RFC stand ard s, relying on p rop rietary extensions to the stand ard MIBs to p rovid e better m onitoring. Exam p les of RMON u se range from gathering sim p le statistics abou t the netw ork like u tilization and collisions, to actu ally cap tu ring p ackets w hich are then retrieved rem otely. There is also SMON , w hich is RMON for sw itched netw orks created by Lu cent, bu t few vend ors other than Lu cent have im p lem ented it. In fact, few vend ors w ho su p p ort RMON bother to fu lly im p lem ent the stand ard p arts of it. They often only inclu d e the first few RMON grou p s and leave the fu ll p acket cap tu ring cap abilities to their p rop rietary extensions. That said , som e u sefu l trou bleshooting d ata can be m ined from RMON . Look into the statistics grou p to d eterm ine if abnorm al traffic is flow ing over a p articu lar p ort . Details m ay inclu d e fram e+ size d istribu tions, u nicast/ m u lticast/ broad cast and fram ing error typ es su ch as ru nts and giants. Calculate epoch time for snmpColD ump, etc. Tim e stam p s for N N M d atabase record s for snmpCollect d ata and for events are kep t as ep och tim e (second s since Janu ary 1, 1970). Place the ep och tim e to translate w here is says “c” below : %OV_BIN%\Perl\bin\perl -e ―print scalar localtime(c)‖ $OV_BIN/Perl/bin/perl -e ‗print scalar localtime(c)‘ 156 14. Notifications N otifications from N N M are generated as action callbacks d efined w ithin the event su bsystem . There are no actions associated w ith SN MP Trap s, so all actions are d efined from the m anager. N ote that m ost N N M events are generated by N N M itself as the resu lts of statu s p olls or other anom alies d etected by N N M‟s d iscovery and p olling engines. The ovactiond d aem on is resp onsible for ru nning actions assigned to ev ents, and actions are configu red in the event configu ration GUI (xnmtrap). Actions associated w ith events are called „action callbacks‟. Configuring an action callback Follow these general step s w hen setting u p an au tom atic action based on an N N M event: 1. Configure and test the script/ tool outsid e of NN M from the com mand line (UN IX: By d efault, the actions ru n w ith UID and GID of user bin) 2. Select useful variable bind ings to pass. The variable bind ings are often listed in the event d escription. For m ost OpenView enterprise events, $2 represents the event source. Ad d itional variables are listed below u nd er Selected Special Variables 3. Configure trusted Cm d s.conf (see below ) 4. Optional step. Test the com m and using the OpenView event OV_application_alert in conjunction w ith the $OV_CON TRIB/ NN M/ send Msg/ send Msg.ovpl script Using trustedCmds.conf file In N N M 6.2+, only com m and s listed in a file in the follow ing d irectory can be execu ted by ovactiond: $OV_CON F/ trusted Cm d s.conf/ Changes to files in tru sted Cm d s.conf m u st be signaled to ovactiond by ru nning the xnmevents –event com m and from the com m and line. tru sted Cm d s.conf is d ocu m ented in the m an/ ref p age for ovactiond. 157 Fognet’s Field Guide to OpenView NNM Any file p laced in that d irectory is p arsed for valid com m and nam es (w ith absolu te p ath) and aliases, for exam p le, su p p ose a file called ringbell in $OV_CON F/ trusted Cm d s.conf contains: ringBell=/opt/OV/contrib/NNM/ringBell/ringBell.ovpl In the "Com m and for Autom atic Action" field (action callback), use “ringBell”. Overrid e the tru sted com m and s featu re by creating a file nam ed ALLOW_ALL in the $OV_CON F/ tru sted Cm d s.conf d irectory. If this file, w hich sh ou ld be em p ty, is fou nd then ovactiond d oes not check w hether a com m and being execu ted is tru sted . General action callback usage considerations Exit cod es. ovactiond listens for the exit cod es retu rned from scrip ts or p rogram s and rep orts a failed action if it d etects a non -zero exit cod e. UN IX UID for ovactiond: ovactiond execu tes com m and s as u ser bin, and thu s is not environm ent or p ath aw are. To change the u ser that ovactiond execu tes as, ad d the –u <user> LRF sw itch in ovactiond .lrf. See p age 5 for the LRF p roced u re. UN IX UID and ping: If the action callback execu tes (or lau nches a scrip t that execu tes) p ing, the com m and m ay p rod u ce a failu re to op en socket. ping m u st be execu ted as the root u ser becau se it op ens a raw socket and only root on UN IX can op en raw sockets. Use the –u root op tion d iscu ssed above. Scalability: When scrip ting or calling p rogram s, it m ay be necessary to consid er issu es w h ich m ay arise from m u ltip le su ccessive calls to ovactiond. Consid er configu ring action s to fork, sp aw n or exec so the child ren d o the w ork and the p arents exit, allow ing ovactiond to lau nch the next bu ffered p rogram calls. Sem ap hore locking m ay also be a u sefu l w ay to d eal w ith overru n issu es associated w ith event flood s. In this case, the first p rocess gets the lock and d oes the w ork. The second p rocess fails to get the lock becau se som eone else is alread y hand ling the issu e, and exits. Also consid er event correlation. The correlation to hand le d u p licate events can be ad d ed very easily u sing the d ed u p .conf (see p age 125), and p revent m u ltip le p ages, for exam p le. For som e correlations like PairWise (p age 124), release of the action m ay be d elayed p end ing the receip t of an event (w hich m ay clear the initial cond ition). See p age 159 below for som e m ore notes on interactions of ECS w ith notifications. 158 Notifications Metacharacters in action callbacks Action callbacks d o not su p p ort Regu lar Exp ressions. The caret (^ ), for exam p le, is u sed to qu ote. The list of m etacharacters is: ; & | < > new-line In ad d ition, any com m and su bstitu tion in the varbind s is also qu oted . Any string enclosed in ' ' or $() is consid ered a com m and su bstitu tion. Selected special variables in action callbacks This is a p artial list of som e of the m ore u sefu l variables: $# $* $n $x $X $V $ar $c $s $$ $E $A $aA $G $S $T The num ber of attributes in the event. All attributes as “seq nam e (type): value” strings. nth attribute as a value string. Must be 1 to 99. Date event received using the local d ate representation. Tim e event received using the local tim e representation. Type: SN MPv1, SN MPv2C, CMIP, or GEN ERIC. The im plied source as an IP add ress. The category the event belongs in. The severity of the event. The $ character. The trap enterprise as a text string if possible. The trap agent ad d ress as d efined in the trap PDU. Sam e as $A except the source as an IP ad d ress. The trap's generic-trap num ber. The trap's specific-trap num ber. sysUptim e in hund red ths of a second since agent up These characters can be sp ecified d irectly in the action callback: a b f n r t v \ ooo xhh Alert (bell) character Backspace Form feed N ew line Carriage return H orizontal tab Vertical tab Shell escape (\ \ ind icates Wind ow s pathnam es.) Octal num ber in range 000-177 H ex num ber in range x00-xFF N otification interactions w ith event correlation After N N M 6.31, certain statu s events m ay be “held ” by the N od eIf and / or Pairw ise event correlations p end ing related events. Au tom atic 159 Fognet’s Field Guide to OpenView NNM actions are held as w ell, and the related events cou ld cau se the child event and its action to be d iscard ed . All p rior versions of N N M alw ays released au tom atic actions im m ed iately. Follow ing that, certain ECS circu its m ay affect the tim ing of the release of au tom atic actions. In p articu lar, the PairWise and N od eIf correlations m ay or m ay not affect the tim ing of the release of au tom atic actions, d ep end ing on the version of N N M. To force au tom atic actions to ru n im m ed iately, regard less of event correlation settings affecting actions, ad d the –f RAW flag to the ovactiond d aem on via the LRF p rocess. ov act iond LRF settings See p age 5 for general instru ctions on m od ifying LRF flags. -u username -t -v -l (UN IX only) set UID and GID for ovactiond N ote: default is bin:bin Trace execution of ovactiond More verbose logging Use logfile other than $OV_LOG/ ovactiond .log -w maxwait Second s to wait for non -zero exit cod e before giving up and killing execution. 0 ind icates never give up; 300 is d efault -f flowtype RAW, CORR, or ALL; see Interactions w ith Event Correlation (above) -s maxlog Maxim um size of logfile; d efaults to 500K Limiting Action callbacks to a subset of objects To set u p an action that ru n s only a su bset of event sou rces, m ake a cop y of the target event, give it a u niqu e nam e, and sp ecify sou r ces u sing “ad d from m ap .” See p age 95 for m ore basics on m anu al event configu ration. “Find ” is a p ow erfu l tool for bu ild ing lists of nod e sou rces based on com m on attribu tes. Once a set of objects is highlighted on the m ap as the resu lt of a “find ” op eration, u se “View - Select H ighlighted ” to select objects across m u ltip le su bm ap s, then “ad d from m ap ” w ithin the Event Configu ration GUI. Selection nam es can be w ild card ed in the event sou rce sp ecification. For exam p le, *sou p * w ou ld m atch selections p easou p and sou p y. IP 160 Notifications Ad d resses can also be u sed as event sou rces, and * or - w ild card s can be sp ecified as w ell, for exam p le: 10.10.0.* or 10.10.0.2-252. See the section below on u sing an external file to sp ecify External File to specify action callback sources To u se an external list of sou rces, enter the fu ll p ath to a text file containing valid selection nam es or IP ad d resses, one p er line, in the sou rces area of the Event Configu ration GUI. Use xnmevents –event to force a re-read of the file after m aking u p d ates. Wild card s cannot be u sed in this external file for either selection nam es or IP Ad d resses. Wild card s can, how ever, be u sed d irectly in the event sou rce sp ecification (see section above). Using Perl for actions callbacks N N M is ship p ed w ith an em bed d ed version of Perl located at $OV_BIN / Perl/ bin/ p erl. Taking ad vantage of this is fine for relatively sim p le scrip ts, bu t it is inad equ ate for scrip ts that call for ad d itional Perl Mod u les becau se N N M‟s Perl d oesn‟t incorp orate all the libraries necessary for extensibility. Install or u se a fu ll-blow n Perl d istribu tion if calling p erl m od u les in scrip ts. For Wind ow s p latform s, ActiveState Perl com es highly recom m end ed . To u se N N M‟s em bed d ed Perl on any p latform , sim p ly u se the “.ovp l” file extension for the scrip t. On UN IX, call Perl at the top of the scrip t u sing: #!/opt/OV/bin/Perl/bin/perl When m aking action callbacks to Perl scrip ts in UN IX, enclose the argu m ents to be p assed in d ou ble qu otes, for exam p le: /opt/OV/bin/myperl.ovpl "$x $X $r $R $*" When m aking action callbacks to Perl scrip ts in Wind ow s, either alias Perl u sing tru sted Cm d s.conf, u se the registered .ovp l if u sing the N N M-em bed d ed version of Perl, or call the fu ll p ath to an alternate Perl execu table, as in: cmd /c start /min c:\\perl\\bin\\perl.exe c:\\perl.pl $2 Action callbacks on Window s platforms The m ost com m on issu e w ith calling scrip ts on Wind ow s p latform s is d ifficu lties in interp reting p ath nam es w ith em bed d ed sp aces and in hand ling sp ecial characters. Use d ou ble qu otes in this case as in the follow ing action callback exam p le: 161 Fognet’s Field Guide to OpenView NNM d:\\‖Program Files‖\\‖HP OpenView‖\\bin\\perl\\bin\\perl d:\\‖Program Files‖\\‖HP OpenView‖\\scripts\\pager.ovpl $r The sim p lest w ay to avoid su ch trou ble is to u se aliases for com m and s via tru sted Cm d s.conf (see the ovactiond m an/ ref). OVSH ELL and OVH IDESH ELL are sp ecial keyw ord s available u nd er N N M on Wind ow s to d isp lay action resu lts in a new w ind ow or to su p p ress the p op u p of a com m and shell. OVSH ELL is the d efau lt. These keyw ord s also w ork w ith aliases d efined in tru sted Cm d s.conf. Exam p le: OVSHELL notepad file.txt (same as ―notepad file.txt‖) OVHIDESHELL cmd.exe /s file.bat OVHIDESHELL telalert –i phil –m ―$2 down‖ In the first exam p le, the p op u p w ind ow is forced . In the second exam p le, an exp licit callback is issu ed that w ou ld requ ire the ALLOW_ALL file exists in tru sted Cm d s.conf. In the third exam p le, the fu ll p ath to the telalert execu table is d efined in tru sted Cm d s.conf and aliased to “telalert.” The equ ivalent to OVH IDESH ELL can be accom p lished w ith: cmd /c start /min command c:\\ bmail -s [email protected] –t [email protected] -f [email protected] -a "$2 down‖ -b "Body text" Or, for fans of Wind ow Scrip ting H ost, p op -u p s can be su p p ressed w ith this exam p le: Set WshShell = CreateObject("WScript.Shell") RetVal = WshShell.Run("CMD.EXE /C dir/w",0,TRUE) Som etim es, it is d esirable to keep the com m and w ind ow u p , as in w hen inp u t is requ ired by the calling p rogram , for exam p le: start /wait c:\\shutdown\\shutdown.exe There is a p roblem w ith u sing p ip es (| ) or red irects (>) in action callbacks on Wind ow s. These are su p p orted on the Wind ow s com m and line, bu t ju st d on‟t w ork in N N M action callbacks. Email notifications using native tools (Window s) IIS is installed w ith N N M. IIS has SMTP cap abilities, w hich ca n be configu red from the IIS p rop erties p age. Once configu red , IIS attem p ts to d eliver files w ith the p rop er form at that are d rop p ed in the follow ing d irectory via SMTP: C:\ Inetpub\ mailroot\ Pickup 162 Notifications A Perl scrip t snip p et that can be u sed as an action callback to send an em ail to a p aging service u sing Microsoft‟s IIS‟s bu ilt-in SMTP server is show n here: $node = $ARGV[0]; # $r. open OUT, ">c:\\page.txt"; print OUT "From: openview\@domain.com\n"; print OUT "To: 12345678910\@archwireless.net\n"; print OUT "Subject: Node $node Down\n"; print OUT "\n"; print OUT "Node $node Down\n"; close OUT; `copy c:\\page.txt c:\\inetpub\\mailroot\\pickup`; `del c:\\page.txt`; exit; MAPISEN D is p art of the Exchange Resou rce Kit. It is on the TechN et CDs. To configu re this, install Ou tlook on the m onitoring server and configu re a p rofile u sing the service accou nt that is being u sed for N N M. H ere is an exam p le MAPISEN D com m and action callback: MAPISEND.EXE -u <outlook profile name> -p <service account password> -r [email protected] -s $2 Down -m ―Node $2 is down‖ Email notifications using native tools (UN IX) mailx and elm are native to both H P-UX and Solaris OS. Both u se very sim ilar syntax w hen invoking from the com m and line. In the follow ing exam p les, the elm and mailx com m and s cou ld be interchanged : echo "$r Up" | mailx -s "$r Up" [email protected] elm -s "$2 down" [email protected] < /tmp/body.txt sendmail can also be u sed , for exam p le: echo 'Subject: Openview Trap object $r for event $N $1 $3\n' | /usr/lib/sendmail '[email protected] [email protected]‘ And yet another w ay for Solaris system s, d em onstrating how to p ass nu ll as the file hand le w here one is requ ired : /usr/ucb/mail -s "$2 down" [email protected] </dev/null Email notifications and D N S issues (UN IX) Som e enterp rise em ail servers d o not forw ard em ail along u nless the send ing system form ats the originating ad d ress as an FQDN . The p reffered m ethod to d o this is by p rop erly configu ring the / etc/ resolv.conf file on the m anagem ent server w ith a line that contains: “search d om ain.com ”. Another m ethod w hich m ay 163 Fognet’s Field Guide to OpenView NNM accom p lish this, if for exam p le N IS is being u sed , is to set the “Dj” setting in the sendmail.cf file. Audio notifications (UN IX) Use $OV_CONTRIB/NNM/ringBell/ringBell.ovpl to ring the bell on the N N M server. To m od ify the volu m e and d u ration of the sou nd , ed it the $APP_DEFS/ XN m events file and find section on ringing the bell. Rem ove the "!" com m ent, set as d esired (try the m axim u m first bellp ercent 100). To ring the bell of any rem ote system that accep ts X d isp lay red irects, inclu d ing PC‟s ru nning X w ind ow s softw are, first create a scrip t “beep .sh” that looks like this: #!/bin/sh echo "This window beeps then closes in 1 second" echo "$1 down" #args passed in order from callback echo ^G #Use real control-G (in vi): CRTL-V CRTL-G sleep 1 N ext ad d the action callback like this: /usr/bin/X11/xterm -display <IPAddr>:0 -e /beep.sh To p lay an au d io file (Solaris only), try: cat crash.au > /dev/audio It m ay be requ ired to change the p erm issions for / d ev/ sou nd to 666 in the / etc/ logind evp erm s file for this to w ork. To enable PC‟s ru nning Exceed X-w ind ow s em u lation, ed it w in.ini and ad d the follow ing: [exceed] DefaultSystemBeep=1 Sound notifications (Window s) Install Wind ow s Med ia Player, or som e other d efau lt WAV file p layer that can be invoked from the com m and line. For the action callback, u se: mplay32 c:\\<path>\\file.wav Example notification setups Exam p le u sing BLAT (Wind ow s): 1. Create $OV_CON F\ trustedCm d s.conf\ testmail containing: testmail=c:\\testmail.bat 164 Notifications 2. Run xnmevents –event from a com m and shell 3. C:\ testm ail.bat contains: C:\blat - -subject "%1 is down‖ -to [email protected] body "%1 is down at %2 on %3 " 4. Action callback for OpenView OV_N od e_Dow n event: OVHIDESHELL testmail $2 $X $x Exam p le u sing Telalert w ith ALLOW_ALL (Wind ow s): 1. Create em pty file: $OV_CON F\ trusted Cm d s.conf\ ALLOW_ALL 2. Run: xnmevents –event from a com m and shell 3. Action callback for OpenView OV_N od e_Dow n event: cmd /c start /min d:\\vytek\\telalertc –i mypager –m ―$2 down‖ Exam p le Perl scrip t that sp aw ns so ovactiond retu rns: 1. Create em pty file: $OV_CON F\ trusted Cm d s.conf\ ALLOW_ALL 2. Run: xnmevents –event from a com m and shell 3. Action callback for OpenView OV_N od e_Dow n event: / usr/bin/perl do_my_thing.pl $A $1 & Popular notification and paging tools The follow ing p rod u cts are often m entioned by N N M end -u sers: Com m ercial Prod u cts: Vend or Prod uct Calam p Sem otus Invoq iTechTool Inventive PageGate Spatch Sysm an Telalert H iplink Alarm point EtherPage Pow erPage N otePage Spatch SMS Server URL w w w.calam p.com w w w.sem otus.com w w w.alarm point.com w w w.ppt.com inventivelabs.com notepager.net w w w.spatch.com w w w.sysm an.no Platform (s) Unix/ Window s Unix/ Window s Unix/ Window s Unix Window s Window s Unix/ Window s Wind ow s Freew are/ Sharew are/ GPL/ Op en Sou rce: Prod uct Send page send page Blat Bm ail qpage Postie URL send page.cpoint.net w w w.send page.org w w w.blat.net beyond logic.org/ solutions w w w.qpage.org infrad ig.com Platform (s) Unix Perl (Both) Wind ow s Wind ow s Unix Wind ow s 165 Fognet’s Field Guide to OpenView NNM Usefu l link for Tap Mod em Ph one nu m bers: http:/ / w w w .notepager.net/ tap -phone-num bers-a.htm Scrolling m essage d isp lays that accep t p ager inp u t: http:/ / w w w .am s-i.com Configuring an attached modem on UN IX servers While som ew hat of a challenge, setting u p a d irectly -attached m od em for p ager notifications can increase the reliability of notification system s that m ay rely on the services (em ail) they are m onitoring as a transp ort for those notifications, thu s elim inating the p ossibility that p ages are m issed if a d ep end ent service (su ch as a netw ork connection to the em ail server) fails. cu (call u nix) can be u sed to in itiate p ages to d irectly-attached m od em s. To u se cu, configu re u u cp to the p ort the m od em is attached to as follow s: Solaris: Set serial port on Solaris / d ev/ cua/ a for a mod em to d ial up: chmod +x /devices/sbus@1f,0/zs@f,1100000:a,cu H P-UX: SAM – peripherals – term inals & m od em s, m od em Ed it / etc/ uucp / Devices Ad d the follow ing line to the end of the file: Direct cul0p2 - 19200 direct Note: The actual cu d evice correspond s to the tty returned by SAM The follow ing com m and can be u sed on Solar is to invoke kermit to send a p age to a locally attached m od em : kermit -l /dev/cua/a -m hayes -b 9600 -C \"set dial timeout 15,dial {$pager_number,,,$message_string #},hangup,exit\" $OV_CONTRIB/NNM/beep95_lx/beep95_lx.sh is a front-end to the cu scrip t, bu t it w as rem oved from later versions of N N M d istribu tions. A cop y can be fou nd in som e search engines. A Perl scrip t can be constru cted to interact w ith cu u sing the op en2 Perl Mod u le, for exam p le in the follow ing snip p et: use IPC::Open2; use Symbol; $WTR = gensym(); $RDR = gensym(); $pid = open2($RDR, $WTR, "cu -s 9600 -b 7 -t -e -l /dev/cua/b"); (code goes here to dial and communicate) print $WTR "~."; # terminate cu close ($WTR); close ($RDR); 166 15. Fault Analysis Tools This section focu ses on tools w hich are p art of N N M (and som e w hich are not) that are u sed to trou bleshoot netw ork fau lts. Problem diagnostics (PD ) PD is inclu d ed w ith N N M AE v7.0+ and is enabled along w ith ET via setu p ExtTop o.ovp l. On initialization, it installs a p robe on the N N M server. Ad d itional p robes can be installed from one of the files p robeH P.tra, p robeSUN .tar or p robeWIN .zip , fou nd in the $OV_Install_Dir/ p d AE/ bin d irectory. V7.53 inclu d ed im p roved d ocu m entation for PD in the Gu id e to Using Extend ed Top ology u ser m anu al. Problem Diagnostics p robes netw ork flow s betw een critical d evices, au tom atically base lining p ath p erform ance. It send s events if p erform ance d eviates from the norm , and d etects p ath blackou ts, brow nou ts, rou ting loop s, flap p ing p aths and other instability. PD u ses tracerou te to d eterm ine Level 3 p aths and u ses ET top ology d ata to p lace Level 2 d evices in p aths. To start/ stop the p robe: UN IX: $OV_MAIN_PATH/pdAE/bin/pdcentral.sh –start|stop DO NOT use the kill -9 com mand on the Java process! Irrecoverable data corruption may occur. Wind ow s: From the Services applet, select NetPath and click Start (or Stop). DO NOT use the Window s Task Manager to term inate the Java process! Irrecoverable d ata corruption may occur. To d isable a p robe: UN IX: Move the file from the rc3.d d irectory. Wind ow s: 167 Fognet’s Field Guide to OpenView NNM Use the Services applet to stop the N etPath service, and run: pdcentral.bat –uninstall To u ninstall p robe and server: UN IX: pdpuninstall.sh Wind ow s: pdpuninstall.vbs To link the p robe to m u ltip le servers: Ed it npprobe.conf on the probe system , m od ify existing lines, then stop and restart the probe. A probe uses the sam e port to talk to all servers. Logs (located in $OV_MAIN _PATH / p d AE/ logs): pd .log npprobe.log m essages from the PD Central application m essages from the probe on this system Trou bleshooting PD: If the GUI applet is not w orking check java console w ithin brow ser for exceptions If PD Central d oesn‟t start via ovstart, try using ovstop pd, then running PD m anually via pdcentral.sh –start or pdcentral.bat –start. Use “<DEBUG>true</ DEBUG>” in the pdconfig.xm l file to generate d ebug output in the pd.log file. This option should only be used briefly because it can generate large amounts of data Use pdcentral.sh –dbmgr (or pdcentral.bat –dbmgr) to launch a UI that allow s SQL queries on the PD d atabase Smart path (access path view ) Sm art p ath is a contribu ted ap p lication ad d ed in V7.51. Sm art p ath has tw o fu nctions: the first is to grap hically show the Layer 2 and Layer 3 access p ath betw een tw o nod es and the seco nd is to find a rou gu e MAC ad d esss. The first fu nction show s u p as a new d ynam ic view called Access Path view . The second fu nction is a com m and line tool. Access p ath view is not enable by d efau lt. Enabling and d isabling Sm artPath w ill stop and restart the ovas p rocess To enable Sm artPath, ru n the follow ing com m and : $OV_CON TRIB/ N N M/ Sm artPath/ ovAccessPath.ovpl –enable The new Dynam ic View is available in the d rop d ow n m enu of N N M‟s H om eBase. This view accep ts tw o end p oints as inp u t. Sm artPath consu lts the N N M Extend ed Top ology and m akes a nu m ber of SN MP qu eries w hich m ay take several m inu tes. If m u ltip le MAC ad d resses are forw ard ed on the last p ort of the last sw itch, an “Unknow n” d evice is ad d ed to the p ath. This hap p ens w hen a hu b or L2 d evice has not been d iscovered by ET. 168 Fault Analysis Tools Find rouge MAC addresses To find the sw itch that a MAC ad d ress is connected to: $OV_CON TRIB/ NN M/ Sm artPath/ ovAccessPath.ovpl –mac <MAC> –file <file> $OV_CON TRIB/ NN M/ Sm artPath/ ovAccessPath.ovpl –mac <MAC> –router <ip> <MAC> is the MAC Ad d ress in qu otes, for exam p le: “00 AA BB CC DD EE”. If the -file p aram eter is u sed , the L2 d evices sp ecified in the file are exam ined to find the MAC ad d ress. The file contents ar e exp ected to be one line p er IP ad d ress. If the -rou ter p aram eter is sp ecified , ET is qu eried for the L2 neighbors of the rou ter, w hich are then exam ined to find the MAC ad d ress. The ou tp u t is an XML d ocu m ent that show s w hich sw itches the MAC is forw ard ed on as w ell as the board / p ort. For each sw itch, a flag is inclu d ed that ind icates if m ore than one MAC is being forw ard ed on the p ort. The sw itch p ort w hich only forw ard s the sp ecified MAC is likely to be the sw itch that the MAC is w ired to. The ping tools The Fau lt-Ping m enu op tion u sed to call the native OS Ping u tility, bu t after N N M 6.2, the natping w rap p er w as ad d ed to convert the p rivate IP into the p u blic IP ad d ress w hen p inging into N AT environm ents. This also changed the tool from u sing a continu ou s p ing to one that only issu es 5 ICMP requ ests then stop s. This is controlled by the “ -n 5” op tion in the natping scrip t in $OV_BIN . Rem ove the op tion in that file to restore continu ou s p ings. Also, to allow the u ser to stop the p ing w hile it is ru nning, change the follow ing line: "system ("$pingExe $PublicIP $pingArgs")" to: "exec ("$pingExe $PublicIP $pingArgs"). In ad d ition, ad d the –followOutput p aram eter to the xnmappmon call to natping in the N N M-IP.tbl registration file in $OV_REGISTRATION / C/ ip m ap . This allow s the end of the ou tp u t to continu ou sly d isp lay so the u ser d oesn‟t have to scroll d ow n to see the resu lts. See p age 6 for m ore info on m od ifying ARF files. 169 Fognet’s Field Guide to OpenView NNM Isolating local netw ork issues The follow ing com m and s can help isolate netw ork p erform ance issu es related to the N N M server, its netw ork stack, and the locally -attached N IC. Check for UDP-related socket overflow s: netstat –p udp Overflow s can ind icate SN MP traffic is overw helm ing the IP This can be becau se of over aggressive d ata collections, or too d evices being p olled . On UN IX, u se ndd to tu ne the netw ork N ote this com m and rep laces the old er nettune com m and u sed H P-UX. stack. m any stack. u nd er rnetstat rnetstat is an N N M-su p p lied SN MP-based version of the fam iliar netstat com m and for p u lling basic netw ork stack d ata via SN MP. rnetstat is also the front-end to som e of the configu ration, fau lt and p erform ance m enu -bar item s in the N N M GUI. Som e of the follow ing rnetstat sw itches are not d ocu m ented or su p p orted by H P: List route table: List connected and listening ports: If nam es, ad d rs, m ask, net add rs: If nam es, status, type, cap., alias: rnetstat rnetstat rnetstat rnetstat –rn <target> –a <target> –In <target> –o ifsum <target> rnetstat –a is a very p ow erfu l tool for retrieving connected p ort info, and w orks by cross-referencing the retu rned MAC ad d resses to the $OV_CON F/ p hysAd d r.conf file. This file can be cu stom ized , bu t is over-w ritten by N N M u p grad es, so take ap p rop riate p recau tions. Som etim es u sers p refer not have the MAC ad d resses translated by the rnetstat com m and . In this case, sim p ly com m ent ou t p articu lar entries in the p hysAd d r.conf file or renam e the p hysAd d r.conf file entirely. D isplay port address mappings There is a m enu -bar item u nd er tools-Port-Ad d ress Map p ing Table w hich d isp lays p ort connections for all d evices connected to the selected nod e. The URL for this tool is: http:/ / <server>:3443/ OvCgi/ connectedN od es?nod e=<nod e> …w here <nod e> is the FQDN of the target nod e and <server> is the nam e of the N N M server. This tool is a front-end for the rnetstat com m and d iscu ssed im m ed iately above. 170 Fault Analysis Tools netcheck netcheck is a hand y bu t p oorly d ocu m ented N N M tool that is the gu ts behind the m enu item : Fau lt ->Test IP/ TCP/ SN MP. It can be issu ed from the com m and line to test one or m ore of the three p rotocols ju st m entioned . Tw o exam p les follow : netcheck -e -o tcpPort=telnet <target> netcheck -e -o tcpPort=21 <target> N ote if the nam e of the service sp ecified , as in the first exam p le above, the tool cross-references the service p ort w ith the services file. mibtable This com m and d u m p s table view s of SN MP MIB su btrees in text form atted ou tp u t, and like rnetstat, is a very p ow erfu l and p op u lar tool for N N M ad m inistrators. For exam p le, to d isp lay p ort connection table entries: mibtable -table ".1.3.6.1.2.1.4.22.1" -fields "Index=1:3,PhysAddress=2:18,IPaddress=3:15" -node <node> On UN IX system s, this exam p le p rod u ces mibtable ou tp u t that is nicely form atted : mibtable -table ".1.3.6.1.2.1.4.22.1" –fields "IPAddress=3,Physaddress=2" -node somenode | nawk '$0~/^[1..9]/ {printf ("%-15s %s-%s-%s-%s-%s%s\n",$1,$2,$3,$4,$5,$6,$7)}' tracert and traceroute These native com m and s, w hich are u sed in N N M‟s Fau lt m enu , show the netw ork hop s a p acket takes to the d estination. The p ath need not alw ays be the sam e, nor m ay be the retu rn p ath. Both com m and s are native OS com m and s. N N M Wind ow s ship s w ith a cop y of tracert.exe in %OV_BIN %, bu t it is no d ifferent that the native one. Wind ow s tracert d iffers from UN IX traceroute in that it is lim ited to u sing ICMP exclu sively (traceroute u ses UDP d atagram s or op tionally, ICMP). Wind ow s also has native u tility called pathping w hich p rovid es broad er p rotocol su p p ort than Wind ow s‟ tracert. With N N M 7.0+, Problem Diagnosis is bu ilt into N N M. This tool p rovid es m ore com p rehensive p athing and statu s d ata if PD p robes are p rop erly configu red . See the beginning of this section for m ore on Problem Diagnosis. 171 Fognet’s Field Guide to OpenView NNM Packet sniffers N N M d oes not contain a bu ilt-in tool for p acket sniffing. H ow ever, som e N N M d aem on tracing facilities can p rod u ce d etailed p acket d u m p s, for exam p le pmdmgr (see p age 131). There are som e com m ercial and freely available p acket sniffers that are w id ely u sed to su p p lem ent the N N M u sers trou bleshooting toolset. Follow ing is a short list of selected sniffers that are u sed by som e N N M ad m inistrators. N etw ork Monitor (netmon.exe) is a p acket sniffer p rovid ed by Microsoft that is available in Microsoft Resou rce CDs. nettl/netfmt is a UN IX native p acket d u m p ing facility that is som ew hat d ifficu lt to w ork w ith, bu t exam p les m ay be fou nd below . snoop is native to Solaris, and its p acket cap tu re form at is the basis of RFC 1761. To cap tu re UDP d atagram s: snoop -x 0 udp <target> Use debug on Cisco rou ters to sniff ou t sp ecific p rotocol issu es. Ethereal from w w w .ethereal.com is free and p orted to all p latform s, and is very p op u lar. N m ap , w hich is available for Wind ow s as N m ap Win is from w w w .insecu re.org. Tcp d u m p , w w w .tcp d u m p .org, is p orted to Wind ow s as WinDu m p . Using nettl nettl is a native tracing u tility on UN IX system s. An exam p le com m and to initialize a nettl trace is: nettl -tn pduin pduout -e <entity> -f /tmp/net w here <entity> is the nam e of the Ethernet entity (d river) being u sed . Get a list of these entities w ith nettl -ss. Typ ically it is BTLAN . A list of the d rivers actu ally u sed on the system can also be obtained u sing: ioscan -kf The nettl com m and p rod u ces trace files nam ed / tm p / net.TRC000 and / tm p / net.TRC001 (circu lar files). In the case w here a w rong entity is selected , the trace file stays w ith a size of 128 bytes. Stop the tracing w ith: nettl -tf -e <entity> The binary trace files can be form atted w ith : 172 Fault Analysis Tools netfmt -l -N -f /tmp/net.TRC00[0,1] To red u ce the ou tp u t, d efine filters (see netfm t m an p age) on a file and u se it on the netfmt com m and line w ith -c <file>. 173 16. Third Party Tools N N M gained p op u larity as an SN MP m anagem ent p latform in large p art becau se of its APIs. Ten years ago, d evice-sp ecific vend ors w eren‟t interested in d evelop ing stand ard SN MP m anager featu res, and H P w asn‟t interested in d evelop ing d evice-sp ecific m anagem ent featu res. That form u la w orked w ell for m any years, bu t it has changed m ore recently. More recently, d evice m akers are less and less interested in p rovid ing m anagem ent tools for even their ow n p rod u cts and H P has resp ond ed by ad d ed d evices-sp ecific m anagem ent su p p ort to N N M. H P m aintains a catalog of certified N N M integrated solu tions at: http:/ / openview.hp.com/ partner/ isv/ ind ex.jsp Using N N M APIs via the N N M Develop er Kit p rovid es access to the N N M SN MP event stream s, allow s cu stom ized top ologies to be au tom atically constru cted , and p rovid es p rogram m atic access to certain N N M legacy d atabase d ata via C++ and JAVA p rogram m ing interfaces. All these sam e p rogram m atic interfaces are also available in Chip Su tton ‟s CS-OV PERL Mod u le available at: w w w .cs-net.com . Below are selected third p arty solu tions, som e of w hich are certified by H P, and som e of w hich are not. This is a sm all list that rep resents those solu tions that are m ost often m entioned by N N M u sers on the Op enView Foru m u ser grou p listserv. Environmental monitoring and contact management Tools for m onitoring room tem perature or for interfacing w ith other analog d evices and then conveying that inform ation to N N M includ e: w w w.im ci.net w w w.netbotz.com w w w.uptim ed evices.com w w w.om nitronix.com w w w.sensorsoft.com w w w.akcp.com Freew are utilities for UN IX Assorted tools used often by other N N M ad m inistrators inclu d e: Expect Sw atch Putty (SSH ) VN C, only better expect.nist.gov swatch.sourceforge.net ww w.chiark.greenend .org.uk/ ~sgtatham / putty ww w.tightvnc.org 174 Third Party Tools Web Load tester I2Trace ww w.w pid alam ar.com/ projects/ 123load test ww w.geocities.com/ m ilicsasa/ Tools/ l2trace Emulating X w indow s on Window s machines Reflections and Exceed are the m ost p op u lar com m ercial tools for em u lating X w ind ow s u nd er Microsoft Wind ow s system s. Freew are tools that accom p lish this inclu d e Xm ing and Cygw in/ X. Users rep ort Xm ing is easier to install and that Cygw in w ith X11 p ackages is m ore highly configu rable. Event correlation tools Tools that su p p lem ent or rep lace the fu nctionality of N N M‟s bu ilt-in event correlation engine, Op enView ECS inclu d e: ECS-based solution provid er SEC – Sim ple Event Correlator InCharge N erveCenter w w w.logec.com kod u.neti.ee/ ~risto/ sec w w w.sm arts.com w w w.openservice.com SN MP and netw ork management tools Tools that su p p lem ent or rep lace the fu nctionality of N N M‟s bu ilt -in SN MP m anager featu res inclu d e : SolarWind s SN MP tools OpenN MS N agios SN MP Toolkit Big Brother Wind ow s Extension agents TrapBlaster N et-SN MP agent MIB Brow ser Bad ger TrapServer w w w.solarw ind s.net w w w.opennm s.org w w w.nagios.org w w w.taave.com w w w.bb4.org w w w.snm p -inform ant.com w w w.realops.com w w w.net-snm p.org w w w.innerd ive.com w w w.bad gerac.com Topology and netw ork modeling tools Tools that aid in top ology m ap p ing and m od eling inclu d e: Ed ge enPortal and nVision Mim ic SN MP Sim ulator Am erigo and other tools Opnet w w w.ed ge-technologies.com w w w.gam bitcom m.com w w w.taave.com w w w.opnet.com Reporting and graphing tools Tools that su p p lem ent or rep lace the fu nctionality of N N M‟s bu ilt-in d ata collections and rep orting featu res inclu d e : RRDtool N MIS Ploticus w w w.rrd tool.com w w w.sins.com.au/ n m is ploticus.sourceforge.net 175 Fognet’s Field Guide to OpenView NNM Cricket OverTim e and PingTim e StatSeeker Kard inia Concord eH ealth MRTG 176 cricket.sourceforge.net w w w.netic.com .au w w w.statseeker.com w w w.kard inia.com w w w.concord .com people.ee.ethz.ch/ ~oetiker/ webtools/ m rtg 17. Cisco Devices This section covers m anaging Cisco d evices w ith N N M. Cisco MIBs The p rim ary reason to load Cisco MIBs is that they contain em bed d ed translations for SN MP trap s that ap p ear in the alarm brow ser. The easiest w ay to integrate Cisco MIBs into N N M is to load the ap p rop riate CiscoWorks-N N M integration p ackage. This also p rovid es cu stom sym bol m ap p ings for som e Cisco d evices. If only the MIB files are of interest, they can be extracted from the integration p ackage and load ed . Look for a file nam ed : AllCiscoMIBs.m y If the integration p ackage is not available, m ost top level Cisco MIBs ship w ith N N M. They can be fou nd in the $OV_SN MP_MIBS/ Vend or/ Cisco d irectory. Load them in d escend ing ord er as follow s: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. SN MPv2-SMI SN MPv2-TC SN MPv2-TM SN MPv2-CON F SN MP-FRAMEWORK-MIB CISCO-SMI CISCO-PRODUCTS-MIB.m y CISCO-TC.m y EN TITY-MIB CISCO-EN TITY-ASSET-MIB The m ost u p d ated Cisco MIB's can be fou nd at ftp :/ / ftp .cisco.com / p u b/ m ibs/ v2/ v2.tar.gz and ftp :/ / ftp .cisco.com / p u b/ m ibs/ v1/ v1.tar.gz 177 Fognet’s Field Guide to OpenView NNM CiscoWorks and N N M For years, CiscoWorks (CW2K) requ ired the p resence of a netw ork m anagem ent p latform su ch as N N M. CiscoWorks then becam e a stand alone p rod u ct w ith a sep arate integration p ackage for sharing d ata w ith N N M. Sim ilar to N N M, CW2K is a collection of integrated toolsets. It is com p rised of Cisco RME (Resou rce Manager Essentials) and CWSI Cam p u s (Cam p u s Manager), w hich com bines CiscoView , Traffic Director (form erly N etScou t RMON ), and ATM Director. In 2007, a nu m ber of other p oint p rod u cts have been ad d ed to the Cisco Works su ite. Also a nu m ber of Cisco netw ork m anagem ent softw are p rod u cts for u se ou tsid e the CiscoWorks su ite have been released . Cisco‟s “Third Party Integration Kit” p rovid es the N N M integration elem ents. This p ackage ad d s CiscoView and Cam p u s Manager to the N N M m enu s, and is intend ed to be u sed for integrating N N M and CW2K w hen both sit on separate p latform s. Both Cisco and H P p rovid e CW2K integration p ackage d ow nload s for N N M. They are su p p osed to be id entical bu t there are ind eed d ifferences. Field consu ltants rep ort few er p roblem s w ith the Cisco-originated integration p ackage and claim the site is kep t m ore u p to d ate. The internal integration u tility that ship s w ith CW2K is intend ed to be u sed for integrating N N M and CW2K w hen both p rod u cts sit on the sam e p latform . Both p rod u cts contain u n -integratable em bed d ed d atabases and CW2K is m em ory-intensive, so it is recom m end ed to ru n CW2K on a sep arate p latform from N N M. If both are to be installed on the sam e p latform , set u p sep arate d isk p artitions for each p rod u ct. The p rod u cts form erly u sed incom p atible versions of JAVA for u ser interface, bu t in 2006, a p atch w as released so CW2K su p p orts JRE 1.4. Once CW2K and N N M are integrated , the recom m end ation is to tu rn off CW2K Availability Polling and setu p N N M to forw ard trap s to DFM. IE brow sers behave best w ith CW2K. See Page 22 for d etails on hand ling m u ltip le version of JAVA clients w hen ru nning CiscoWorks and N N M on the sam e client or server. CiscoWorks, N N M, JPI and Internet Explorer CiscoWorks 2000 requ ires Java Plu g-In 1.3.1, and w hen a JPI 1.4.1 ap p let is started from the sam e instance of Internet Exp lorer, you can get Internet Exp lorer errors. If you start CiscoWorks 2000 from ovw, 178 Cisco Devices the system w ill try to u se the sam e brow ser w ind ow that w as last lau nched . If that brow ser w as ru nning H om e Base (and hence JPI 1.4.1), Internet Exp lorer w ill throw a ru ntim e error. If you first start CiscoWorks 2000, then lau nch H om e Base, you w ill see a d ialog box, "Attachm ent to a ru nning Virtu al Machine failed ", follow ed by an Internet Exp lorer crash. The w orkarou nd is to start CiscoWorks 2000 in a sep arate Internet Exp lorer instance. Start a new Internet Exp lorer (from the start m enu , not u sing File: N ew ) and brow se to http :/ / <m achine>:1741. This w ill start CiscoWorks 2000. Start another new Internet Exp lorer w ind ow , then start H om e Base from the ovw m enu . This w ill u se the new IE w ind ow , and leave the CiscoWorks 2000 ap p let in the old w ind ow . Alternatively, you can start a new IE w ind ow after starting H om e Base, so that CiscoWorks 2000 w ill start in its ow n instance of Internet Exp lorer. N N M and handling Cisco VLAN s By d efau lt, Cisco sw itches store the CAM table for VLAN 1 in the d ot1d Tp Fd bTable, w hich is the table N N M‟s d iscovery u ses to bu ild level 2 top ologies. If there is nothing in that table, i.e. d ot1d BaseN u m Ports retu rns 0, then N N M m ap s it as a “flat” top ology. This can be overcom e by d iscovering the d evice w ith the com m u nity string ap p end ed w ith @vlan_no w h ere vlan_no is a VLAN in u se on the sw itch. This can cau se trou ble for d iscovery if qu erying an N N M u nsu p p orted d evice. Be su re to u se the latest Cisco Agents for the N N M version u sed . See Page 219 for the URL to check ET d evice agents. Cisco bu ild s sep arate brid ge tables for each VLAN . To view MIB d ata for VLAN s, issu e snm p get or snm p w alk requ ests w ith the VLAN nu m ber ad d ed to the com m u nity string, follow ed by the @ character. For exam p le, the com m u nity p u blic@200 gives in fo on VLAN 200‟s MIB d ata. N ote that VLAN 1004 is p resent on every Cisco sw itch (d efau lt VLAN for FDDI). To get a list of VLAN s try one or the other of: snmpwalk vtpVlanState.1.1 <node> snmpwalk.1.3.6.1.4.1.9.9.46.1.3.1.1 <node> To get m ore granu lar VLAN traffic rep orts from Catalyst 6000 sw itches, setu p N DE w ith a netflow collector and configu re brid ged featu res su ch as brid ged flow stats. N DE d etails can be fou nd at: 179 Fognet’s Field Guide to OpenView NNM w w w .cisco.com / univercd / cc/ td / d oc/ prod uct/ lan/ cat6000/ sw _7_6/ confg_gd / nd e.htm Another ap p roach is m ining the vlan.d at d atabase file from the sw itch. This file contains d ata p ertaining to IDs, nam es, p ort m em bership , tru nks, VTP, etc. This is only configu ration d ata, how ever. If ru nning a version of IOS less than 12.1(18.4)E or 12.2(20.4)S there w as a bu g w here if the VLAN table w as em p ty, the d evice w ou ld generate au thentication trap s. ET w ill p oll for VLAN s. N N M’s Cisco discovery configuration feature N N M V7.51 introd u ced the Cisco Discovery Configu ration (CDC) featu re to help control and im p rove the sp eed of ET Discovery. This p ow erfu l scalability featu re allow s the d ata p u lled from these d evices to be controlled w ith resp ect to the IP Ad d ress and / or the SN MP OID. Discovery of the follow ing entities can be controlled w ith CDC: • • • • • Forw ard ing Database (FDB) Tables Per-Mod ule (Per-Board ) Inform ation for Multiple Board s Interface to Board Mapping VLAN Mem bership Inform ation Port Aggregation The CDP Agent com es w ith an m si-file for Wind ow s installation. Enable CDC by cop ying the CiscoSw itchSnm p .u ser.cfg file from $OV_N EW_CON F/ H POvCiscoAgt/ conf/ nnm et/ agents/ to the $OV_CON F/ nnm et/ agents d irectory and m od ify the new cop y. Before enabling CDC, note that w hile CDC can im p rove p erform ance, it m ay d o so at the exp ense of accu rate top ology d ata and CFA d ata that is u sed by the APA. It m ay be beneficial to d isable the Forw ard ing Database (FDB) tables on strategic d evices. Doing so w ill force reliance on only on CDP for connectivity, bu t connectivity to non -CDP d evices from Cisco d evices w ill then not be d iscovered . For exam p le, if you have a Cisco sw itch that is connected to a Wind ow s server, you w ill not d iscover the connection to the server even thou gh the server is m onitored by N N M. If d isabling the FDB tables is not p ractical, see the section below on setting ET to give CDP p riority over FDB. H P recom m end s exp erim enting on a few test d evices before d isabling FDB tables m ore w id ely. They su ggest m onitoring the “N u m ber of L2 links” stat in the “View Top ology Statu s” tab u nd er “Discovery Statu s” 180 Cisco Devices after ap p lying changes that affect CDC. When configu ring CDC, it is very im p ortant to read and u nd erstand the H P White Pap er on CDC: $OV_DOC/ WhitePapers/ CiscoDiscoveryConfiguration.pd f In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. Setting ET to prefer CD P over FD B connection data If ET is enabled , ET w ill create m u ltip le connections on an interface w hen a connection is rep orted both in CDP and FDB for the sam e interface. N N M V7.51 Interm ed iate Patch 18 introd u ced a m ethod to force ET to p refer a CDP connection over an FDB connection in case m u ltip le connections are being rep orted from the sam e interface on a Cisco d evice. To enable this behavior, configu re the m _UseInferenceLogic flag w ith a valu e of 17 in the follow ing file: $OV_CON F/ nnm et/ DiscoSchema.cfg file. Useful Cisco IOS SN MP commands To d isable Cisco TCP connection reset trap s generated w hen a telnet session to a Cisco d evice is reset: snmp-server disable traps tty See “Configu ring Cisco IOS Trap s” below for m ore on this. To force a Cisco d evice to TFTP its config file, u se snmpset on w riteN et (.1.3.6.1.4.1.9.2.1.55); it is equ ivalent to being logged on the Cisco d evice and issu ing a 'w rite netw ork'. For exam p le: snmpset –c <WriteCommunityString> <RouterName> .1.3.6.1.4.1.9.2.1.55.192.168.1.1 octetstringascii <TargetFilename> The target d irectory m u st exist and have p rop er file p erm issions. To set the sou rce ad d ress of SN MP trap s to be the loop back IF: snmp-server trap-source loopback 0 The com m and to send a test SN MP trap , for exam p le, the chassisAlarm On trap (trap nu m ber 6, sp ecific nu m ber 5): test snmp trap 6 5 181 Fognet’s Field Guide to OpenView NNM H ere are the com m and s to set rou ter to check avgBu sy1 every 60 second s and send s trap s on rising/ falling threshold s of 70/ 60 p ercent: rmon event 1 trap RM7net description "High CPU Util" rmon event 2 trap RM7net description "Low CPU Util" rmon alarm 1 lsystem.57.0 60 absolute rising-threshold 70 1 falling-threshold 60 2 owner <system> To d isable link u p / link d ow n from non -IP interfaces on a sw itch: conf t int gi 1/1 no snmp trap link-status Assign or change ifAlias for an interface. Generally, ifDescr and ifN am e are not w ritable. N ote that Cisco CAT OS d oesn‟t su p p ort ifAlias, and w rites instead to the p ort nam e in the p ort table u nd er the Cisco Stack MIB. conf t int fa0/1 description <newname> ifAlias MIB is: m ib-2.ifMib.ifMibObjects.ifXtable.ifXentry.ifAlias Advanced Board status via the APA The APA p oller p rovid es board entity statu s and generates sp ecial events relating to board s if p rop erly configu red . See p age 79 for m ore inform ation on this featu re. Command to create cut SN MP view s The exam p le w ou ld cu t the rou ting table from being view ed by a server nam ed hp ov. This w ou ld keep N N M from consu m ing too m u ch CPU on rou ters w ith very large rou te tables, like ed ge rou ters. This can also be accom p lished by setting the “-R” lrf op tion on netmon (see the section on lim iting d iscovery on p age 39). More on this in the p ap er referred to below on the u nd ocu m ented CPU p riority sw itch. snmp-server view hpov internet included snmp-server view hpov ip.21 excluded snmp-server community <comm-string> view internet RO Hex target IP for Cisco ping MIB Use the follow ing PERL cod e to covert an IP Ad d ress to hexad ecim al notation for setting the CiscoPingAdress octetstringhex using the MIB Browser or using snmpset commands: 182 Cisco Devices #! /opt/OV/bin/Perl/bin/perl # convert IP address to hex @i = split ('\.',$ARGV[0]); $hip .= sprintf("%02X", $_) foreach (@i); print "$hip\n"; Undocumented IOS commands The u nd ocu m ented snm p -server com m and is u sed to set SN MP CPU p riority: snmp-server priority {low | normal | high} The “Q” colu m n of “ show process” show s cu rrent p riority This d oes not affect SN MP trap p rocess. For m ore, see: w w w.cisco.com / w arp/ public/ 477/ SN MP/ ipsnm phighcpu.shm l Use the follow ing u nd ocu m ented event logging control com m and for toggling syslog event m essages: logging event {link-status | subif-link-status} The “no” form of the logging event link-status <interface> com m and is u sed to tu rn off send ing u p , d ow n and change m essages for an interface to the syslog. This is very u sefu l on live system s since these system s generate so m any of these m essages. This is a com p anion com m and to the d ocu m ented com m and w hich p revents send ing the associated snm p trap : no snmp trap link-status. Weird trap OID s (.1.3.6.1.6.3.1.1.5.4.1.3.6.1.4.1.9) This is an SN MPv2 origin event OID. To resolve, create an OID alias in trap d .conf. For a fu ll exp lanation, see the section on origin event OIDs on p age 107. Unknow n trap 1.3.6.1.2.1.0.1&2 from Catalyst 2950 The Cisco 2950 Catalyst sw itch IOS has a bu g that cau ses the u nknow n trap . Up grad e the IOS. H ere is the text of the Release N otes for IOS 12.1_(9)EA1: “When enabling traps on a catalyst 2950, m essages are received at t he N MS w ith erroneous object id entifiers (.1.3.6.1.2.1.0.2 and .1.3.6.1.2.1.0.1)” 183 Fognet’s Field Guide to OpenView NNM Cisco linkD ow n trap configuration Som etim es, the TRAP m acros for Cisco MIBs d on‟t p rop erly d efine the Cisco OID_ALIAS p rop erly, so Cisco Link d ow n trap s show u p in the alarm brow ser w ith a m essage like: linkDown trap received from enterprise cisco.1.469 with 4 arguments The solu tion is to ad d an enterp rise id entification u sing the event configu ration GUI for the .1.3.6.1.4.1.9.1.469 enterp rise. Alternatively, m anu ally ad d the enterp rise to the $OV_CON F/ C/ trap d .conf file in OID_ALIAS section, for exam p le: OID_ALIAS ciscoProductsC2621 .1.3.6.1.4.1.9.1.469 Ru n xnmevents -event after ap p lying d irect tr ap d .conf ed its. Controlling Cisco link dow n trap variable bindings With IOS version 12.1, Cisco ad d ed su p p ort for RFC 2233 conform ance. This ad d ed su p p ort gives end -u sers a choice for w hat sort of d ata is being p assed w ith the linkDow n and LinkUp trap s. RFC 2233 d efines linkUp and linkDow n trap s in the Interfaces Grou p MIB m od u le (IF-MIB.m y) as follow s: linkDown NOTIFICATION-TYPE OBJECTS { ifIndex, ifAdminStatus, ifOperStatus } ::= { snmpTraps 3 } linkUp NOTIFICATION-TYPE OBJECTS { ifIndex, ifAdminStatus, ifOperStatus } ::= { snmpTraps 4 } Prior to this change, Cisco‟s link d ow n/ u p trap s conform ed w ith RFC 1573, as d efined in the Cisco Interface Cap ability MIB m od u le (CISCO IF-CAPABILITY.m y) as follow s: VARIATION linkUp -- TRAP-TYPE -- OBJECTS { ifIndex, ifDescr, ifType, locIfReason } VARIATION linkDown -- TRAP-TYPE -- OBJECTS { ifIndex, ifDescr, ifType, locIfReason } The ad vantage of RFC 2233 conform ance is that SN MP is su p p orted for su b-interfaces. The d isad vantage is that the ifDescr, locIfReason, and ifTyp e are not inclu d ed . Those objects m ay be m ore d esirable d ata to have in the event m essage (for notifications, etc.). N ote that if u sing the old er behavior, locIfReason shou ld be an arbitrary valu e for su binterfaces. The Cisco IOS com m and s to control this behavior are below : 184 Cisco Devices Set RFC 2233 conform ance: Set the old er behavior: snmp-server trap link ietf no snmp-server trap link ietf Interface index remapping N N M‟s m ore recent versions have featu res to d etect interface ind ex rem ap p ing that m ay occu r after d evice reboots, etc. These rely on u se of ifAlias, w hich requ ires consistent interface nam ing p ractices. ET and the APA also have facilities to hand le ind ex re-m ap p ings, bu t these algorithm s are not p erfect. The follow ing Cisco IOS com m and locks ou t ind ex rem ap p ings and at a m inim u m shou ld be ru n on d evices that are being p olled by N N M‟s d ata collector to facilitate consistent rep orting: snmp-server ifindex persist ifSpeed and polling issues w ith data collections Cisco's locIf*BitsSec are gau ge variables, exp onentially d ecaying average com p u ted as: being a 5 m inu te (previous m inute rate) + (the previous 5 m inute rate) ───────────────────────────────── 2 Unlike variables like ifInOctets, w hich are p u re cou nter valu es, the p olling interval has no effect on the valu es stored by snm p Collect. ifSp eed is a MIB-2 variable, and m ay not alw ays contain accu rate d ata for the interface for Cisco d evices. Unless the sp eed is m anu ally configu red in the rou ter, it shou ld be the d efau lt, w hich is T1/ E1 for any serial connection, 10M for eth, 100 for fa, etc. If there are su binterfaces configu red , they inherit the sp eed of the u nd erlying p hysical p ort. With Fram e Relay, if the rou ter is connected to a sw itch, the rou ter takes clocking from the sw itch, ignoring any configu red sp eed setting. An incorrect ifSp eed configu ration w on't affect transm ission in this case, bu t it m ay cau se errors for the rou ting p rotocol in u se. From an N N M p ersp ective, m ost u tilization collections and p erform ance grap hs m ay show false d ata since som e of them u se ifSp eed to calcu late u tilization. Interesting Cisco MIBS, OID s and MibExprs Use CISCO-CON FIG-MAN -MIB.m y to track config changes Use CISCO-ISDN -MIB.m y for basic rate ISDN on access rou ter 185 Fognet’s Field Guide to OpenView NNM Use CISCO-VPDN -MGMT-MIB-V1SMI.m y for VPDN Per-u ser Cisco MIB's for Layer 2, CDP, and VLAN : cisco-stack.m ib: portIfInd ex, vlanPortMod ule, vlanPortVlan cisco-cd p.m ib: cd pCacheAd d ressType, cd pCacheAd d ress cd pCacheDevicePort cisco-vtp.m ib: vlanTrunkPortManagem entDomain cisco-vlan-m embership.m ib: vm Vlan Cisco system stats are m ostly fou nd u nd er the lsystem MIB tree. CPU bu sy p ercentage in the last 5 second p eriod . N ot e that these m easu rem ents are based on the last 5 second p eriod in the sched u ler, not the last 5 realtim e second s. This m akes them better ind icators of p otential issu es that arise from CPU consu m p tion: .1.3.6.1.4.1.9.2.1.56 --busyPer .1.3.6.1.4.1.9.9.109.1.1.1.1.3 1 m inu te exp onentially-d ecayed m oving average of the CPU bu sy p ercentage: .1.3.6.1.4.1.9.2.1.57 --avgBusy1 .1.3.6.1.4.1.9.9.109.1.1.1.1.4 5 m inu te exp onentially-d ecayed m oving average of the CPU bu sy p ercentage: .1.3.6.1.4.1.9.2.1.58 --avgBusy5 .1.3.6.1.4.1.9.9.109.1.1.1.1.5 m em %util "Percent available Mem ory on a Cisco Device Com puted by: ((TotalDRAM – freeMem )/ TotalDRAM) * 100”\ .1.3.6.1.4.1.9.3.6.6.0 \ .1.3.6.1.4.1.9.9.48.1.1.1.5.2 \ + .1.3.6.1.4.1.9.9.48.1.1.1.6.2 \ + .1.3.6.1.4.1.9.2.1.8.0 \ .1.3.6.1.4.1.9.3.6.6.0 \ - .1.3.6.1.4.1.9.9.48.1.1.1.5.2 \ + .1.3.6.1.4.1.9.9.48.1.1.1.6.2 + / 100 * PIX Firew all failover statu s: .1.3.6.1.4.1.9.9.147.1.2.1.1.1.3 cfw H ard w areStatusValue variable from CISCO FIREWALL-MIB. It could be active(9) or stand by(10), d epend ing on the prim ary unit's status. 186 Cisco Devices What is exponentially-decayed moving average? Com m on in stock charting as w ell as p erform ance m onitoring, this typ e of correlated statistical averaging is also know n as exp onentially w eighted or exp onentially d am p ed m oving averag es (EWMA), and is com m only d ep loyed in Cisco SN MP agents. This statistical m od el allow s for good com p ensation for lost d ata p oints or w ild sp ikes com bined w ith the need to keep very few p ast sam p les in m em ory. In essence, this m od el gives the highest w eigh ts to the m ost recent d ata p oints, p rovid ing a good ind ication of general volatility. A concise and straightforw ard exp lanation of how Cisco im p lem ents it can be fou nd at: w w w .cisco.com / w arp / p u blic/ 66/ 3.htm l Cisco temperature probes N ot all Cisco d evices have variables that p ertain to environm ental m onitors are instru m ented the sam e. For Catalyst 6000 series sw itches, load the follow ing MIBs: CISCO-EN TITY-SEN SOR-MIB, EN TITY-MIBV1SMI. The OID for entSensorValu e is: .1.3.6.1.4.1.9.9.91.1.1.1.1.4. Each entSensorValu e instance in the above variable m ap s to an entPhysicalInd ex (1.3.6.1.2.1.47.1.1.1.1.2) for w hich an entPhysicalDescr (1.3.6.1.2.1.47.1.1.1.1.2) is need ed . Som e Cisco d evices u se the EN VMON MIB, inclu d ing Catalyst 5000‟s, 7x00 rou ters 12000 rou ters, etc. ciscoEnvMonTemperatureStatusIndex: 1.3.6.1.4.1.9.9.13.1.3.1.1 ciscoEnvMonTemperatureStatusDescr: 1.3.6.1.4.1.9.9.13.1.3.1.2 ciscoEnvMonTemperatureStatsValue: 1.3.6.1.4.1.9.9.13.1.3.1.3 ciscoEnvMonTemperatureThreshold: 1.3.6.1.4.1.9.9.13.1.3.1.4 ciscoEnvMonTemperatureLastShutdown: 1.3.6.1.4.1.9.9.13.1.3.1.5 ciscoEnvMonTemperatureState: 1.3.6.1.4.1.9.9.13.1.3.1.6 N ote that 3600 rou ters only rep ort ciscoEnvMonTem p eratu reState. D irectly integrating Cisco syslog messages A lightw eight version of H P‟s Op enView Op erations (OVO) logfile encap su lator agent w as integrated into UN IX-only AE versions of 187 Fognet’s Field Guide to OpenView NNM N N M 7.0 and later. This facility p rovid es a robu st syslog encap su lation tool for p assing syslog entries to N N M alarm brow ser. This facility is intend ed only for “occasional” u se, for exam p le for traversing firew alls for few critical d evices w here SN MP cannot p ass. Becau se of the ad d ed system overhead , it is not recom m end e d for u se as a rep lacem ent for SN MP trap s, w hich are m u ch m ore efficient and scalable. See p age 142 for d etails on the N N M syslog facility. Cisco-specific event correlation circuits in N N M The Event Classifier circu it classifies Cisco events into m eaningfu l categories grou p ed by d evice. The chassis failu re circu it m onitors Cisco trap s for tem p eratu re, fan failu re and p ow er su p p ly fau lts , consolid ating a set of environm ental tra p s into new Op enView Enterp rise events. Cisco links MIBs: w w w .cisco.com / public/ sw -center/ netm gm t/ cm tk/ m ibs.shtm l Mibs via FTP: ftp:/ / ftp.cisco.com / p ub/ m ibs/ oid / Mib Repository and trap translator: w w w .cisco.com / cgi-bin/ Supp ort/ Mibbrow ser/ unity.pl Configuring Cisco IOS Traps: w w w .cisco.com / en/ US/ tech/ tk648/ tk362/ technologies_tech_note 09186a0080094a05.shtm l Mib supported by VPN 3000 concentrator: ftp:/ / ftp.cisco.com / p ub/ m ibs/ supportlists/ vpn3000/ vpn3000supportlist.htm l Configuring VPN 3000 concentrator to send events as SN MP traps: w w w .cisco.com / univercd / cc/ td / d oc/ prod uct/ vpn/ vpn3000/ 3_6 / config/ events.htm Syslog H ow -to: w w w .cisco.com / en/ US/ prod ucts/ sw / cscow ork/ ps2073/ p rod uct s_tech_note09186a00800a7275.shtm l OSPF Configuration Managem ent w ith SN MP: w w w .cisco.com / en/ US/ tech/ tk869/ tk769/ technologies_w hite_pa per09186a00801177ff.shtm l RAN CID (Really Aw esom e N ew Cisco config Differ): 188 Cisco Devices (freew are for autom ated login to Cisco d evices) w w w .shrubbery.net/ rancid / DFM White Paper: w w w .cisco.com / w arp/ partner/ synchronicd / cc/ pd / w r2k/ d vftm n/ prod lit/ d igd f_w p.htm CiscoWorks links CiscoWorks Docu m entation w w w .cisco.com / univercd / cc/ td / d oc/ prod uct/ rtrm gm t/ cw 2000/ H P-based N N M Integration Ad ap ter: w w w .openview .hp.com / d ow nload s/ try_nnm _0001.htm l Cisco-based N N M Integration Ad ap ter (CCO login requ ired ): w w w .cisco.com / kobayashi/ sw -center/ cw 2000/ cm c3rd .shtm l and w w w .cisco.com / cgi-bin/ tablebuild .pl/ cw 2000-u tility CiscoWorks in Large-Scale environm ents White Pap er: cisco.com / w arp/ pu blic/ cc/ p d / w r2k/ prod lit/ ckspp_w p.htm Extend ed SAA agent su p p ort for MPLS/ VPN s: w w w .cisco.com / en/ US/ prod ucts/ sw / iossw rel/ ps1839/ prod ucts_ feature_guid e09186a0080087b0a.htm l#w p 1037443 189 18. ovw Map Operations ovw is th e foregrou n d p rocess th at lau n ch e s th e N N M static m ap . Un d er UN IX, it is som etim es called th e Motif GUI. It m ay be r eferr ed to as th e legacy GUI in fu tu re ver sion s sin ce H P p lan s to d ep recate th at old er in terface in fav or of H om ebase, or JAVA -based d yn am ic view s. IPMAP is th e p rogr am th at r u n s to bu ild th e ovw 5-tiered top ology based on OSI layer 3. ovtopmd is th e backgr ou n d p rocess th at feed s statu s to IPMAP. IPMA P is th e ap p lication th at is registered w ith th e ovw APIs to p r od u ce th e fam iliar lev el 3 top ology -based v iew th at is broken in to a h ierar ch y of five su bm ap layers: Root, In tern et, N etw ork, Segm en t, an d N od e. Con tain m en t realm s a d d v irtu al layers at th e In tern et an d N etw ork levels to allow u ser s to logically cu stom ize th eir top ologies. IPMAP reads data from the map database and synchronizes it with data in the topology database via ovtopmd. netmon provides direct status to interfaces at the node level and the APA provides direct status via the status bridge ovet_bridge to ovtopmd. Map operations – selecting, cutting and pasting objects Selecting m u ltip le objects can be tricky for m ap op erations for cu tting, cop ying and p asting. If care is not taken, inad vertent cop y/ p aste or cu t/ p aste op erations can be irreversible and u nrecoverable . Som etim es, in ord er to recover certain m ap cu stom izations, d atabase w ip es and netw ork red iscovery m ay be necessary. The follow ing keyboard op erations are su p p orted : Multiple select: Multiple select: Multiple select: Move selected : Cut/ copy – paste: Cut/ copy – paste: 190 Control key and click on objects Rubber band select Rubber band and control-click d eselect Left click and d rag Right click popup m enu H otkeys ctrl-x/ c/ v – use w ith care ovw Map Operations Tips for using the find operation Most find op erations key off of N N M object d atabase field valu es. To p eru se the available field valu es for a p articu lar object, Right -click over a nod e in the ovw GUI and select object p rop erties. Any of the object p rop erties su ch as cap ability flags can be u sed in search op erations. Find op erations can also be m ad e based on sym bol p rop erties, so all nod es that m ap to a certain sym bol typ e can be selected . A very hand y w ay to select a bu nch of objects that are in m u ltip le su bm ap s (to cop y and p aste to a container, for exam p le), is to ap p ly a find op eration w hich highlights to search resu lts then select Ed it >Select H ighlighted . Exact m atch of the string entered in the find field is the d efau lt behavior for find op erations. N ote the exact m atch rad io bu tton in the find w ind ow . Regu lar exp ressions (RegExp ) are su p p orted in find op erations: . Any printable character * ^ $ \ [ ] Zero or more of the preceding character If first, following chars match beginning If last, following chars match string at end Escape special characters Ranges of characters Exam p les of find strings: 192\.168\.[1-4]\.[0-255] ^.*\.bc\.ca All IPs in subnets 192.168.1 through 192.168.4 Any nod e in the domain: bc.ca Setting unset capability fields Cap ability field s are u sefu l for find op erations, for filtering op erations, and for bu ild ing rep orts u sing ovtopodump or ovobjprint. Cap abilities that are u nset are not d isp layed . Change this behavior by m od ifying the $APP_DEFUALTS OVw (see p age 9) setting and changing the valu e of the below keyw ord to TRUE: show UnsetCapabilitiesField s TRUE 191 Fognet’s Field Guide to OpenView NNM Map object visual cues Figure 18-1. A pplication Plane: User Plane: Highlighted: Selected: Executable: Transparent: The IPMAP ap p lication p laced the object Object can‟t be p laced logically in IPMAP The resu lts of a Find op eration The objects can be p assed to actions Dou ble click on the object lau nches an action as op p osed to op ening an u nd erlying su bm ap Object has bitm ap , bu t no “box.” This is a d evelop er featu re u sed by third p arties Summary of ov w submap properties The follow ing attribu tes can be set for a su bm ap : - Show Connection Labels - Subm ap persistence - Background Graphics - Layout (auto, bus, star, ring, row / colum n, PTP, etc.) - Wind ow Geom etry (size and placem ent of subm ap) - Overlay (on| off, explod e into sam e or new w ind ow ) 192 ovw Map Operations N ew objects do not show up in maps When u sing loadhosts (see p age 45) or w hen new objects show u p in the object d atabase, as seen via ovobjprint, they som etim es cannot be fou nd in any m ap s. A com m on reason this occu rs cou ld be that the netw ork or the segm ent that the nod e is a p art of is u nm anaged . Use find on the d evice‟s su bnet to confirm that the netw ork or segm ent exists and is m anaged . Tw o devices combine into a single node There are several reasons for this, and several things to check w hen this hap p ens. First, confirm that the DN S nam es assigned to all interfaces on the tw o nod es have no overlap s. And , if u sing DN S, also check if there is anything in / etc/ hosts, since it m ay be looking there after failing to find an entry in DN S. N ext, check that the d evices d o not have any interfaces that have the sam e IP ad d ress assigned . If this w ere the case, a d u p licate IP ad d ress alarm shou ld have been generated . Du p licate IP ad d resses are not su p p orted u nless they m eet the requ irem ents d iscu ssed u nd er OAD on p age 246. If an IP ad d ress had been first assigned to one nod e, then re-assigned to the other nod es, there cou ld be shad ow d ata that N N M is p icking u p t hat cau ses it to m erge the nod es. Im p rop er su bnet m asks on the interfaces cou ld cau se the d evices and N N M trou ble in attem p ting to learn the new assignm ent d ata. In ord er to resolve this situ ation, d elete the nod es from the top ology and op en any closed N N M m ap s. Ru n: xnmsnmpconf –clearC. Search the N N M d atabase u sing all the nam es associated w ith both nod es. Use m enu bar Find -Object by Selection N am e. H ighlight and d elete any retu rned resu lts and let N N M red iscover the nod es. See p age 45 for inform ation on u sing loadhosts. If objects p ersist and continu e to tu rn u p in object find resu lts, ru n : ovtopofix as d escribed in the the p roced u re on p age 19. Ru n: snmpwalk on the interface tables of the d evices. On Cisco d evices, w alk the Ip Ad EntAd d r table. In som e cases, d evices w ill retain old IP ad d resses w here they shou ld n‟t and rep ort these long after an interface has been re-IP‟d . Using the IOS global com m and "clear interface nam e/ nu m ber" w ill in m ost circu m stances clear the IP ad d ress. If it is a d eleted virtu al interface, reconfigu re the interface, assign an u nu sed IP ad d ress, clear the interface, and then d elete the 193 Fognet’s Field Guide to OpenView NNM interface again. If none of these m ethod s w ork, a rou ter reboot w ill in m ost cases clear the stale entries from the SN MP tables. Controlling trunked or meshed connections in ov w Mu ltip le connections betw een objects on a m ap are rep resented as m etaconnections, w hich are connections that exp lod e to vie w the u nd erlying m u ltip le connections. If it is p referable to rep resent these as a single connection via u ser-sp ecified sp ecific p orts, this can be accom p lished u sing the $OV_CON F/ netm on.equ ivPorts configu ration file. N ote that p rior to V7.51 Interm ed iate Patch 18, netm on cou ld n‟t su p p ort m u ltip le p ort equ ivalence record s for a single host, bu t that p atch resolved that and m u ltip le entries can now be m ad e in the file. Understanding symbol and object delete operations The key d istinction to help u nd erstand d elete op erations is that from the GUI p ersp ective, sym bols are d eleted , not objects. From the d atabase p ersp ective, objects are only d eleted if all sym bols rep resenting that object are rem oved from all m ap s. Objects d eleted from one m ap w ill not be rem oved from the object d atabase if there are sym bols rep resenting that object in another m ap . The “synchronizing” m essage that ap p ears w hen a m ap is op ened is the op eration that keep s the m ap d atabase and the top ology d atabase in sync. This op eration reconciles objects that are m arked for rem oval in the object d atabase w ith sym bols rep resenting them in that m ap ‟s m ap d atabase. If there are no m ore sym bols left corresp ond ing to the u nd erlying object, it is rem oved from the object d atabase. "Delete From This Su bm ap " only d eletes sym bols from that one su bm ap . For exam p le, a gatew ay object m ay exist in m u ltip le segm ent‟s su bm ap s. "Delete From All Su bm ap s" m arks for d eletion every instance of that sym bol from all su bm ap s, and all sym bols that are contained if the sym bol is a container object. The right-click m enu d elete is equ ivalent of "Delete from all Su bm ap s." Finally, the follow ing com m and externally d eletes an object: ovtopofix –r <node name> 194 ovw Map Operations Fatal IPMAP or ovw init failed errors These errors typ ically ind icate at least one corru p t ed object in the Map , Top ology, or Object d atabase. The p roced u re im m ed iately below typ ically resolves the issu e. If not, it m ay be necessary to contact H P su p p ort. It m ay also be necessary to d elete and rebu ild the d atabases then red iscover the netw ork (see p age 12). Find ing the offend ing object m ay not be that easy, how ever. Usu ally, the ou tp u t of rep eated invocations of ovtopofix and ovw –mapcount (u sing the op tions su ggested below ) can help id entify the object that is cau sing issu es. Remove a stubborn object from all maps and databases Typ ically this is requ ired w hen an object is corru p ted becau se its configu ration has d rastically changed and N N M hold s stale inform ation abou t the object. Before an object can be d eleted from the top ology and object DBs, it m u st be d eleted from all m ap s. Objects are not d eleted from m ap s u ntil they are op ened . If an object is d eleted from one m ap , it is m arked for rem oval in the object DB and shou ld be rem oved from m ap s that su bsequ ently op ened d u ring the m ap synchronization p rocess. Follow these step s to m ake su re that objects are rem oved (in highly-scaled environm ents, rem ove the –v op tions to increase p erform ance): 1. 2. 3. 4. 5. 6. 7. 8. 9. Delete object using “d elete from all subm aps” option in ovw Open all m aps and let them synchronize; close all maps Run: ovstop netmon; then run ovw –mapcount –cruDRv Repeat #3 until no errors are reported Run: ovtopofix –chs0v Run: xnmsnmpconf –clearCache, then start netmon, ovw Confirm object is rem oved from DBs using find from ovw or by running: ovtopodump –RISC <object> If object is still in DB, run: ovtopofix –r <object> Red iscover d evice using: loadhosts (see page 45) Remove “REMOVED ” objects from databases Typ ically, op ening all m ap s and ru nning the above p roced u re take s care of m ost situ ations. Still, objects m arked for rem oval m ay still p ersist. In this case: 1. Select "Locate->Objects->By Selection Nam e" and enter REMOVED in the "Regular Expression" field then hit "Apply" or "Enter" 195 Fognet’s Field Guide to OpenView NNM 2. 3. 4. This find s all REMOVED objects and "H ighlights" them on the m ap Click "View ->H ighlights->Select H ighlighted " N ext click: "Ed it->Delete->From All Subm aps" If the REMOVED objects still p ersist in the d atabase, stop netmon and ru n the follow ing com m and s: ovw -mapcount -RuD ovtopofix -a xnmsnmpconf -clearCache Manually add objects The com m and loadhosts (see p age 45) exists to au tom atically force the d iscovery of objects that netmon cannot d iscover by itself. loadhosts is typ ically attem p ted w hen netmon fails to d iscover d esired objects. Use this p roced u re w hen loadhosts itself fails. A typ ical exam p le is a d evice (p erhap s in a DMZ) that is on the other sid e of a firew all from the N N M server. Com m only, the firew all p erm its SN MP traffic, bu t blocks incom ing or ou tgoing ICMP traffic or both. A sim ilar scenario is often seen w hen u sing MPLS or IPVPN tu nnels/ clou d s w here access to an ISP's d evices is restricted . It is easy to ad d an object for w hich the ap p rop riate segm ent -level su bm ap alread y exists. Sim p ly choose Ed it-Ad d Object. It is trickier w hen the segm ent or the netw ork sym bol above it d on‟t exist. Before attem p ting su ch op erations, search the top ology u sing “find ” for the IP su bnet to m ake su re it not hid d en from view or p laced in a hard to find container in the top ology. When ad d ing a netw ork object into the internet m ap , be su re to choose the “IP N etw ork” netw ork sym bol from the Ad d Object p alette; IPMAP w on‟t recognize any other sym bol typ e at this top ology layer. Give it a label and note that the “Ad d Object” d ialog has the “IP Map ” Object Attribu tes available, and not grayed ou t. If it is grayed ou t, som ething is am iss. It‟s im p ortant to select and set the “IP Map ” attribu tes. Minim ally, set the netw ork N am e, Ad d ress, and Su bnet m ask. All su bsequ ent entries into this object‟s su bm ap m u st m ap to these attribu tes in ord er to be recognized by IPMAP. A “verify” shou ld retu rn: “OK.” Once the netw ork object is ad d ed , it shou ld not show u p in the u ser p lane. 196 ovw Map Operations In ad d ition, it is a good p ractice to u se / etc/ netw orks or you r DN S to enter inform ation abou t the netw ork, and the u se the -m op tion to loadhosts to set the correct netm ask. N ow , op en the netw ork object and create a segm ent object , w hich also m u st be configu red w ith the p rop er “IP Map ” attribu tes su ch as su bnet m ask. Op en the segm ent object, then ad d a generic nod e d evice, w ith the “IP Map ” attribu te m inim ally containing the IP ad d ress, and then d em and p oll it. Externally manage or unmanage objects Version 6.31 introd u ced the –g and –G op tions to the ovotopofix com m and . Earlier version on N N M p ast v ersion 6.0 p rovid e su p p ort for the -g and -G op tions via p atches. Version 6.41 introd u ced the follow ing Dynam ic View s URL‟s w hich allow objects to be m anaged or u nm anaged rem otely: http:/ / <N N M-Server>::7510/ topology/ manage http:/ / <N N M-Server>::7510/ topology/ unmanage all N N M versions, the DIM com m and : xnmtopoconf – manage/unmanage can be u sed to externally m anage or u n m ange For d evices, bu t this only w orks w ith AE version of N N M. Overriding IPMAP symbol changes IPMAP has rather narrow logic abou t m ap p ing constru cts. When a sym bol is created by IPMAP, it is consid ered "m anaged " by IPMAP, and any su bsequ ent sym bol ed its are su bject to IPMAP's review for correctness. When netmon issu es a configu ration p oll, any changes to su ch an object m ay be reset by IPMAP. For exam p le, if the sym bol is u p d ated w ith a p referred nam e, the nam e m ay change back to the original nam e that IPMAP chose for it. Use the follow ing environm ent variable setting to p reserve m anu al settings of IPMAP objects: IPMAP_NO_SYMBOL_CHANGES=TRUE Or, set the “-u ” op tion to the IPMAP ap p lication call in the IPMAP registration file. See p age 5 for m ore on the ARF p rocess. See p age 3 for m ore on setting environm ent variables. 197 Fognet’s Field Guide to OpenView NNM Map status propagation rules Defau lt: This is d efined by the num ber of sym bols that are norm al, com pared to the num ber of sym bols that are not norm al. Ad m inistrative status values are ignored (Unm anaged , Testing, Restricted , Disabled .) The calculation is: Unknown: N ormal: Warning: Minor: Major: Critical: N o Norm al or abnormal sym bols All sym bols norm al One sym bol abnormal; m ultiple n orm al: Multiple sym bols abnorm al; m ultiple norm al One sym bol normal; all others abnorm al All sym bols are abnormal Prop agate Most Critical This setting sim ply takes the m ost critical status and propagates it from the interface to the nod e, from the nod e to the segm ent, from the segm ent to the netw ork, and from the netw ork to the internet sym bol in the root subm ap. Customizing map status propagation rules The u ser can d efine the p ercentage of sym bols in a given state that cau se that state to be u sed . If tw o states are satisfied then the m ost severe state shou ld be u sed . H ere are the d efau lt valu es for this op tion : >5% >10% >20% >30% >0 else Critical Major Minor Warning N ormal Unknown For exam p le, if 25% of the sym bols are Minor and 15% are Major then Major is the statu s that is u sed , since it is the m ore severe statu s. Symbol status ovw su p p orts three d ifferent w ays in w hich color ( statu s ) can be set on a sym bol: "Com p ou nd Statu s", "Sym bol Statu s" and "Object Statu s". These op tions allow ap p lications and / or IPMAP control over the color of the sym bol. To d eterm ine or change a sym bol‟s statu s, sim p ly right click over the sym bol and select "Sym bol Prop erties". Com p ou nd Statu s Com pound status is used w hen the parent sym bol w ants to reflect the com bined status of the sym bols in the sym bol's subm ap. e.g. Networks, Segm ents etc. Sym bol Statu s Sym bol Status is used w hen an application w ishes to have control over the color of sym bol that may d iffer d epend ing on w hich subm ap the sym bol is on . 198 ovw Map Operations Object Statu s ovw propagates the status the sam e w ay to all the sym bols in the m ap that represent this object. The value of this status is stored in the object in the map. Symbol status influences Statu s is stored in the m ap d atabase as w ell as in the top ology d atabase. Map synchronization reconciles d ifferences. IPMAP u ses the follow ing ru les to assign statu s sou rce : - Interface sym bols are set to object status source. - Devices in segm ents and networks are set to sym bol status. - Segm ents and networks are set to com pound status. - Routers in internet subm ap are set to com pound status. N ote that this ru le su ggests that rou ter sym bol on a netw ork su bm ap u ses sym bol statu s, w hile the sym bol on the internet su bm ap u ses com p ou nd statu s. This m eans the rou ter sym bol on the netw ork su bm ap reflects only the interface statu s of the rou ter in that netw ork, how ever the statu s reflects all the rou ter's interfaces on the internet su bm ap . See the ―-s compoundNodeOn‖ sw itch (d iscu ssed below ) to change this behavior so all sym bols reflect the m ore global statu s of the d evices. Fine tuning ov w symbol status IPMAP has op tions that allow som e interesting levels of granu lar control w ith resp ect to statu s that goes beyond setting sim p le statu s p rop agation ru les. The ―-s‖ sw itch to IPMAP affects a m ap behavior, bu t only at the initial creation tim e of the m ap , in the follow ing w ays: Connections reflect the status of und erlying interface: -s connStatusOn Connections d o not reflect the status of und erlying interface: -s connStatusOff Status of nod e sym bols reflects the status of all com ponents. This m eans that sym bols on m ultiple levels and subm aps representing the sam e object reflect m ore consistent status: -s compoundNodeOn 199 Fognet’s Field Guide to OpenView NNM Transient, persistent and on-demand submaps To im p rove scalability, H P ad d ed the ability t o sp ecify the levels of the IPMAP top ology w hich are load ed into m em ory u p on start u p of the ovw m ap in version 5.0. This is called the „on-d em and su bm ap featu re‟. A transient su bm ap is one that is not load ed into m em ory and a p ersistent su bm ap is one that is. Transient su bm ap s take longer to load from the GUI and their contained objects are not available to ovw API actions su ch as the ability to change their statu s p rogram m atically. The p ersistence filter w as im p lem ented to allow filter-based ru les to d eterm ine w hat su bm ap s are load ed . Persistent filters are m ostly need ed by a few third p arty p rod u cts that requ ire their objects being in m em ory. In highly-scaled environm ents, they can also be u sed to keep su bm ap s containing p articu lar d evices in m em ory for qu icker GUI access. By d efau lt, all levels are p ersistent for UN IX installations. For Wind ow s installations, only the internet level and the contain m ent realm s are p ersistent. Persistent levels are set throu gh m ap op tions in the ovw GUI or via the “-d level” sw itch op tion to the ip m ap callb ack in the $OV_REGISTRATION / C/ ip m ap ARF registration file. See the IPMAP m an/ ref p age for sw itch level valu es and p age 6 for m ore on u p d ating ARFs. Care is need ed , thou gh, in m aking higher IPMAP levels transient. For exam p le, rou ter sym bols on the internet su bm ap w hich w ou ld norm ally take "com p ou nd statu s" cou ld p otentially not p rop erly u p d ate their statu s since there m ay not be a nod e su bm ap in m em ory from w hich to calcu late its statu s. N ote that only p ersistent su bm ap s are inclu d ed in m ap snap shots. setStatus utility This contribu ted u tility sets the "object statu s" valu e of a m ap sym bol. For setStatus to w ork, the sym bol rep resenting the object m u st have its statu s sou rce set to “object”. Also, the ovw m ap m u st be op en in ord er to change the statu s of the object. This m eans that a setStatus 200 ovw Map Operations issu ed on an object w hen the m ap hap p ens to be closed is not su bsequ ently changed w hen the m ap is eventu ally op ened . The p ath to the scrip t is: $OV_CONTRIB/NNM/setStatus/setstatus.ovpl In ord er to m anu ally set a sym bol to allow its statu s to be changed by the scrip t, right click on the icon and choose Sym bol Prop erties. Change Statu s Sou rce to “Object.” Occasionally, an error m ay be generated w hen th e object sou rce d oesn‟t p rop erly m ap to the selection nam e of the target of the setStatus com m and : Could Not set status color for XXXX - Object not on map This error can be su p p ressed by setting the follow ing $APP_DEF xnm events resou rce (see p age 9 for m ore info): warnOnUnknownSource:false If setStatus is u nw orkable and insu fficient for externally m anip u lating an object‟s statu s, it is very easy to change the statu s sou rce for an object or a set of objects u sing the N N M APIs. See p age 174 for m ore on the ovw API. Lining up symbols in a perfect line or row Perfectly lining u p sym bols u sing m anu al d rag op erations is very d ifficu lt. Another m ethod is to exp ort the m ap , then ed it the exp ort file, m od ify the X and / or Y coord inates in the file to m atch the line w hich the sym bols fall on, then im p ort the file back into the m ap . N ote that ed iting the exp ort files is not su p p orted and cou ld resu lt in m ap d atabase corru p tion if syntax errors are introd u ced throu gh ed iting. Back u p the m ap d b p rior to attem p ting to ed it exp ort files. See p age 273 for the p rop er backu p p roced u re. Forcing icons to scale dow n in size Consid er that each su bm ap is a d efau lt p ixel w id th and height. That d efau lt afford s the largest icon that is registered . Things tend to get clu ttered w hen there are a lot of icons are on a su bm ap . So, to m ake the icons scale d ow n in size, the d efau lt p ixel w id th and height for the su bm ap need s to be m ad e larger w hile keep ing the w ind ow size the sam e. 201 Fognet’s Field Guide to OpenView NNM To d o this, tu rn au to layou t on. If there are any backgrou nd grap hics, they have to be tem p orarily rem oved becau se OV locks the p ixel w id th and height to the backgrou nd . N ow , d rag the icons to the extrem e left, right, top , and bottom of the w ind ow and notice that au to -layou t au tom atically shrinks the icons as the p ixel w id th and height exp and . Keep rep eating this p rocess u ntil the icons are sm all enou gh to get the nu m ber of nod es d esired onto the su bm ap . Once this is com p lete, au to layou t can be tu rned back off, backgrou nd grap hics can be re-ap p lied . At this p oint, save the geom etry for the su bm ap in the View m enu . Forcing icons to scale up in size In general, tu rning au to layou t off and m anu ally sp read ing ou t the icons m ay cau se the m ap to red raw the icons at a larger p ixel size. See also the section above on lining u p icons to sp ecific XY coord inates. The follow ing $APP_DEF p aram eter can be u sed to set the m inim u m icon size for m ap s w ith m any objects: OVw *layoutMinSym bol Rad ius This setting is u sefu l w hen it is im p ortant to visu ally d isp lay the color changes on very bu sy m ap s. Details on u p d ating $APP_DEF p aram eters can be fou nd on p age 9. Finding and closing open ov w maps To d eterm ine how m any and w hich ovw m ap s are op en, ru n: ovstatus –v ovuispmd To d eterm ine w hich u sers have m ap s op en (UN IX only), ru n the follow ing com m and , w hich m ay show u sers of the d efau lt m ap : fuser -u $OV_DB/openview/mapdb/default/current/map.lock To stop all ovw sessions: ovstop ovuispmd IPMAP tracing Assign the nam e of a logfile to the follow ing environm ent variable and IPMAP tracing is logged to that file w hen ovw is started in that environm ent: 202 ovw Map Operations IPMAP_LOG_FILE See p age 3 for d etails on setting environm ent variables. N etw ork named “10” or “arpanet” on internet submap N N M nam es netw ork sym bols by u sing the nam e fou nd in / etc/ netw orks or ..\ system 32\ d rivers\ etc\ netw orks. The d efau lt version of that file fou nd in an u nm od ified state on som e UN IX system s (p articu larly H P-UX) has only the “10” netw ork d efined w ith "arp anet" as the historical nam e. See p age 111 for a d iscu ssion of this issu e. The solu tion m ay be as sim p le as changing the d efau lt entry in / etc/ netw orks, then d eleting and red iscovering the d evices in the affected netw ork. If a nam e for the corresp ond ing su bnet nu m ber isn't fou nd in / etc/ netw orks, then N N M w ill nam e the netw ork sym bol as the su bnet nu m ber. N ote that N N M rem oves trailing zeros from the nam e. Therefore, 10/ 8, 10.0/ 16, 10.0.0/ 24 and 10.0.0.0/ 30 m ight all look like "10" w hen nam ed . It m ay be a good id ea to assign d ifferent nam es in the netw orks file to these netw orks if they exist. See p age 27 for m ore d etails on how netw ork objects are nam ed . Also, avoid having any / 8 / 22 netw orks in you r netw ork. N U0 interfaces turn red in map In som e instances, nu ll interfaces are u sed for rou te ad vertising aggregation, and they m ay be im p ortant d ep end ing on how the netw ork is setu p . In m ost cases, these interfaces can safely be u nm anaged . If there are m any, globally filtering them ou t u sing the follow ing p roced u re m ay be an op tion: If using the APA poller, set up an ifType filter to unm anage them (see page 86); if using netmon, use ovautoifmgr (see page 42). In either case, the affected nod es should be rediscovered , or use netm on.interfaceN oDiscover (7.5+), w hich is used by both the netm on and APA pollers. Symbols superimposed on each other in submap This is a bu g in N N M 6.2 and it w as fixed w ith a p atch . The p roblem w as seen w hen d oing cu t-and -p aste op erations into su bm ap s w ith au to-layou t tu rned off. This only affected only read -only m ap s. 203 Fognet’s Field Guide to OpenView NNM Managing VLAN s A d iscu ssion on m anaging VLAN ‟s for Cisco d evices can be fou nd on p age 179. In general, VLAN SN MP info is virtu alized in the Brid ge MIB via the SN MP com m u nity string. For exam p le, to list MAC ad d ress for a VLAN for vtp VlanState.1.3, w here 3 is the VLAN nu m ber: snmpwalk -c passwd@3 <switch> .1.3.6.1.2.1.17.4.3.1.2 ET‟s VLAN View p rovid es su p p ort for a lim ited nu m ber of sw itches and p rotocols. Cisco VLAN s are d iscovered via the brid ge MIB, the Cisco Stack and CDP MIBs. Other vend ors that p rovid e good su p p ort for VLAN View d iscovery inclu d e Enterasys, Extrem e, and H P. Som etim es, interconnected sw itch es w hich carry VLAN traffic that are not on the m anagem ent VLAN can cau se the N N M top ology segm ent to change, d ep end ing on w hich sw itch is p olled . In this case, the netm on.equ ivPorts file can be u sefu l in hard cod ing to p ort connections from the N N M p ersp ective. See p age 194 for m ore inform ation on u sing netm on.equ ivPorts. Also, assu re you have the m ost u p -to-d ate ET d evice agents for you r critical connector d evices. See Page 219 for m ore on d evice su p p ort. Managing VPN endpoints H ow N N M m anages VPN ‟s d ep end s on how the VPN behaves. If the VPN is alw ays stable and p resent, it ju st looks like an IP tu nnel. Even if there is at least one d evice on the other sid e of the tu nnel that is p ingable and SN MP su p p orted , it p robably w on‟t be d iscovered by netmon, bu t in this case, the reachable d evices can be load ed into the top ology via loadhosts (see p age 45). Other d evices on the rem ote su bnet m ay be d iscovered natu rally after d em and p olling the first p ingable, SN MP-su p p orted object, bu t they also m ay not. ARP u sage (requ ired for N N M to au to-d iscover) is not stand ard across VPN s. Again, u se loadhosts to force everything to be d iscovered . If the VPN is not stable, those links m ight be m anaged in an ind irect m anner, p erhap s via VPN -related enterp rise MIBs u sing an SN MP d ata collection that generated threshold alarm s based on a change in a p articu lar MIB table entry. Or, p erhap s the u p stream d evice m ight su p p ort VPN -sp ecific trap s. Another p ossibility is that the u p stream d evice can be configu red to log im p ortant m essages w ith the syslog facility (see p age 142). 204 ovw Map Operations Managing ISD N interfaces Polling d orm ant backu p ISDN links m ay cau se them to activate. The best p ractice is to set u p an ACL in the rou ter to block certain typ es of traffic that cou ld activate them , i.e. ICMP, SN MP, RIP, etc. Most vend ors‟ SN MP agents, inclu d ing Cisco, rep ort inactive ISDN interfaces as “d orm ant” in the ifOp erStatu s MIB2 table. Setting the “ d ” oid _to_typ e flag (see p age 7) also restricts statu s p olling (via netmon) to these interfaces. Observe that this flag w ill not restrict p olling of d ow nstream d evices on the LAN segm ent connected to the nod e w ith the ISDN interface. Sw itching routers and routing sw itches in ov w Also called m u lti-layer sw itches, there is an increasing p resence of chassis w ith su p p ort for board s that rou te and board s that sw itch. N N M‟s ovw m ap s have a hard tim e m ap p ing sw itches that contain rou ter m od u les or rou ters that contain sw itching m od u les. This is becau se those d evices often d on‟t consolid ate SN MP fu nctions. For IPMAP top ology in general, level 3 d ata com es from RFC 1213 MIB2 tables and level 2 d ata com es from RFC 1493 brid ge MIB tables. Extend ed Top ology w orks d ifferently in this resp ect and m ore d etails on that can be fou nd on Page 229. Som e of these d evices su p p ort sep arate SN MP agents for the rou ter m od u le and the sw itch m od u le and m ay not be able to be ad d ressed p rop erly via SN MP u nless sep arate IP ad d resses are assigned to the d iscrete m anagem ent agents. In som e cases, the d evice m ay be rep resented as tw o sep arate objects in the ovw m ap s, p articu larly if sep arate nam es have been assigned to the m anagem ent ad d resses in DN S. This m ay be d esirable d ep end ing on the netw ork ad m inistrator‟s p references. If this is not d esirable, u se DN S A record s for the uniqu e ad d resses, bu t also assign CN AME record s that are the sam e - then netmon can resolve m u ltip le ad d resses as being aliased to the sam e nam e. The objects m ight have to be d eleted and red iscovered after su ch a change. ET is better at rep resenting these d evices. In N N M 7.5, there is su p p ort in ET for board -level d iscovery and m ap p ing, and this greatly help s w ith this sort of d evice. While ovw m ap s m ay still have d ifficu lty p rop erly rep resenting these d evices, the ET rep resentations shou ld be ju st fine w ith this new er cap ability. 205 Fognet’s Field Guide to OpenView NNM One issu e, thou gh, is that APA p olling m ay be read ing d ata from one d iscreet agent or the other and the agent being p olled is not the d esired agent. netmon d iscovery p icks a p referred SN MP ad d ress and the APA u ses this ad d ress. See p age 44 for the p roced u re w hich exp lains how to change the p referred SN MP ad d ress. Wireless N N M started su p p orting w ireless netw ork d iscovery in V6.4 (ET 2.0). Wireless access p oints are d iscovered by netmon via SN MP and N N M assigns the d evices the “isWireless” ovwdb attribu te. Tuning ov w db for large numbers of objects If ovobjprint -S show s m ore th an 5000 objects, u se the -n LRF flag to ad ju st the d atabase to the p rop er size for the nu m ber of objects being m anaged . The nu m ber shou ld be set to 5-10% above the nu m ber of objects rep orted . See p age 5 for the LRF p roced u re. Limiting ov w menu bar access for some users The basic p rocess to lim it m enu bar access is to cop y the registration tree to another location, p are d ow n the registration files in the cop ied location by searching for the registration file entries that are to be rem oved , then set the OVw RegDir environm ent variable to p oint to the m od ified registration tree d irectory in the u ser login shells for the u sers w ho are invoking the m od ified trees. To set environm ent variables, see the p roced u re on p age 3. On Wind ow s, be carefu l to set %OVw RegDir% in all u sers' p rofiles or logon scrip ts, esp ecially the d efau lt p rofile for new u sers. Limiting usage of read-w rite maps N orm ally, the first instance of m ap is op ened in read -w rite m od e; su bsequ ent m ap s are op ened in read -only m od e. Only one read -w rite instance (ovw session) of a m ap can be op en at any one tim e. The $OV_CON F/ ovw .au th file w as introd u ced in N N M 6.0 and is the p referred m ethod for control u ser access to m ap s. This d oes not p rovid e som e of the granu larity of control offered by the below m ethod s, how ever. Ad d the follow ing OVw ap p -d efau lts settin g to change the d efau lt m ap op en state to read -only: OVw*readOnly: true For instru ctions on how to m od ify ap p lication d efau lts, see p age 9. With this setting, read -w rite m ap s can be op ened only by 206 ovw Map Operations exp licitly u sing the –rw op tion to the ovw com m and . Another m ethod for forcing u sers to be able to only op en read -only m ap s is to alias or hard cod e the ovw com m and in the u sers shell accou nt (UN IX), to the ovw –ro com m and . Sim ilar tricks can be p layed in Wind ow s environm ents. N ote that u sers w ho invoke ovw w ith the –rw op tion get read -only m ap s if the m ap p erm issions d on‟t p erm it access via ovwchown or ovwchmod com m and s (UN IX only). Limiting access to certain N N M applications N N M V7.51 Interm ed iate Patch 18 introd u ced the ovp rocess.allow configu ration file in $OV_CON F to allow som e p rocesses to have lim ited u ser-based access by u ser id . The ap p lications this ap p lies to in V7.53 inlcu d e: xnmsnmpconf ovalarmadm xnmevents xnmtrap Manipulating map ow nership and permissions ovwperms is a front end to the follow ing three u tilities: ovwchown ovwchgrp ovwchmod allow s maps to associated w ith UN IX users allow s maps to associated w ith UN IX groups sets m aps w ith UNIX-like perm issions These com m and s are not su p p orted u nd er Wind ow s. Any attem p ts to try to d u p licate ovwperms-like activities on Wind ow s system s, for exam p le trying to change ACL‟s on the N N M DB files or d irectories, cou ld resu lt in d atabase corru p tion. N ote the com m and ovwls, w hich m ay be fam iliar to u sers of old er N N M versions, is d ep recated in later versions of N N M. ovw .auth, ovw db.auth, ovspmd.auth, and ovserver files The ovw .au th, ovw d b.au th , and ovsp m d .au th files w ere introd u ced in N N M 6.0. They control, resp ectively, u ser access to the ovw GUI, client access to ovwdb for rem ote consoles, and u ser access to the ovstop , ovstart and ovstatu s com m and s. With N N M 6.0, the $OV_DB/ op enview / ovw d b/ ovserver file w as introd u ced as w ell. This file is sim p ly a text file w ith the N N M hostnam e and need s to be checked w hen restoring d atabases from backu p s from hot stand bys, w hen hostnam es change, and w hen m igrating N N M to new hosts. 207 Fognet’s Field Guide to OpenView NNM Window s-specific monitoring tools in N N M Menus Wind ow s N T Menu op tions are not actu ally d efined in any registration files. These m enu entries seem to be generated by a p rogram called ovwNTtools.exe. The execu tion of this file, how ever, is itself d efined in a registration file, nam ely $OV_REGISTRATION / N Ttools. Move this file ou t of the registration tree to d isable these Wind ow s-sp ecific m enu bar op tions. See p age 5 for d etails on the LRF Proced u re Add right-click pop-up menu items to ov w GUI The follow ing ARF ad d s a right-click p op -u p m enu item that issu es a ping to the target nod e. Also, see notes on natping on p age 169. Application "Popup_Ping" { Description { "HP OpenView", } DisplayString "Ping"; /* Use -Shared to make sure only one instance of the process is executed at any one time. */ Command -Shared "xnmappmon"; PopupItem <100> "Ping" Context (AllContexts || isIP || isRouter || WantPopupMenus) TargetSymbolType ANY f.action "natping"; Action "natping" { MinSelected 0 ; SelectionRule (isNode || isInterface) && isIP; #ifdef NT CallbackArgs "-helpBrowser nnm:netPingTsk\ -commandTitle \"Ping\" \ -appendSelectList – ` appendSelectListToTitle \ -cmd Perl/bin/perl -S natping "; #else CallbackArgs "-helpBrowser nnm:netPingTsk\ -commandTitle \"Ping\" \ -appendSelectList – appendSelectListToTitle \ -cmd natping "; #endif } Ad d the above into a file w ith any nam e u nd er the $OV_REGISTRATION / C d irectory and check the syntax w ith the com m and : regverify –arf <filename>. A restart of ovw sessions is then requ ired . Add menu item to launch SSH on selection (UN IX) See above for fu ll p roced u re and u se this ARF snip p et in the action block of the registration file: Action "ssh" { MinSelected 1; 208 ovw Map Operations MaxSelected 1; Command "/usr/dt/bin/dtterm -title "${OVwSelection1}" -e ssh ${OVwSelection1}&"; } Another ap p roach is to call a scrip t and p ass it a sp ecific u sernam e that d iffers from the d efau lt, w hich in the above ARF w ill be the u ser w ho op ened the ovw session. Also, su ch a scrip t cou ld not requ ire a selection nam e so a host can be p rovid ed if a nod e is not selected . In this case, sim p ly rem ove the “MinSelected ” requ irem ent from the ARF. Su ch a scrip t m ight look like this: #!/bin/sh stty erase ‗^H‘ target=$1 if [ ―$target‖ = ―‖ ] then /usr/bin/echo ―Enter host to connect to: \c‖ read target echo ―‖ fi /usr/bin/echo ―Enter Username: \c‖ read logon /usr/bin/ssh $logon@$target /usr/bin/echo ―‖ 209 19. Web Interface The Web GUI is based on legacy ovw GUI top ology and only p rovid es visibility to legacy IPMAP top ology. This interface is to be d istingu ished from the Dynam ic View s JAVA interface d iscu ssed in Section 21. The legacy w eb interface requ ires an op en ovw session to be ru nning and the u ser sessions are alw ays read -only view s. In the p ast, the Web GUI w as a CGI-based ap p lication. With N N M 6.4, Jakarta Tom cat becom es the op erative servlet. URLs N ote that the UN IX Web GUI access p ort changes from 8880 to 3443 in N N M Version 6.4 Web GUI: http:/ / nod enam e/ OvCgi/ ovlaunch.exe (Window s) http:/ / nod enam e:3443/ OvCgi/ ovlaunch.exe (UNIX) Lau nch a sp ecific m ap nam e : http:/ / nod enam e:port/ OvCgi/ jovw .exe?MapNam e=MyMap Lau nch a sp ecific m ap su bm ap nam e : http:/ / nod enam e:port/ OvCgi/ jovw .exe?MapNam e=MyMap&ObjectNam e =MySubm ap Alarm Brow ser: http:/ / nod enam e:port/ OvCgi/ ovalarm .exe Rep orter: http:/ / nod enam e:port/ OvCgi/ nnm ReportPresenter.exe N N M V7.51 Interm ed iate Patch 18 ad d ed the follow ing ad d itional URL op tions w hich are d ocu m ented in the N etw orkPresenterURLop tion.p d f w hite p ap er in the $OV_DOC/ WhitePap ers d irectory: ServerName, objectName and PreferredSubmapType, for exam p le: 210 Web Interface http:/ / hostnam e[:port]/ OvCgi/ jovw .exe?[MapNam e= <mapname>&Object N am e=<selectionname>&ServerNam e=<servername>(Preferred Subm apType =<submaptype> | Subm apN am e=<submapname>)] Maintaining open ov w map sessions for w eb clients Web clients requ ire op en ovw sessions. (n ote that Dynam ic View s d o not requ ire op en ovw sessions). There are creative w ays to m ake su re ovw sessions are available to serve w eb clients w ithou t having to d ed icate a w orkstation to m aintaining ovw sessions for this p u rp ose. xvfb: The UN IX X Virtu al Fram e Bu ffer can be u sed to su p p ort ovw sessions. xvfb is free and su p p orted by the X consortiu m and can be d ow nload ed from w w w .x.org. Once installed , set u p an rc file to au tom atically lau nch a session on system startu p . Don‟t forget to set u p a kill scrip t as w ell. Make su re to give the nu m bered startu p link file a higher nu m ber than the one u sed by the Op enView d aem on startu p rc file. In the scrip t, enter: /usr/bin/X11/Xvfb :10 & DISPLAY=:10 ovw & Or, a slightly m ore verbose version : /usr/X11R6/bin/Xvfb :10 -screen 0 1152x900x8 2>> /var/adm/Xvfb.log& VN C: VN C (also know n as REALvnc) is only one of m any virtu al w ind ow ing p rod u cts. VN C is free, m u lti-p latform , and w id ely u sed in the u ser com m u nity. It can be fou nd at: w w w .vnc.com Also consid er a m ore actively-d evelop ed and stable VN C d erivative, tightVN C: w w w .tightvnc.com . To configu re an ovw session to start w hen the system starts u p or w henever ovstart is issu ed , ad d an LRF file for ovw (d etails on LRF p roced u re on p age 5): 1. 2. 3. 4. Create a LRF file, named ovw.lrf w ith the follow ing lines: ovw:ovw: OVs_YES_START:ovuispmd:ro:OVs_NON_WELL_BEHAVED:15: Register the new LRF via: ovad dobj ovw.lrf N ote the –ro option makes the session read -only. Map options can also be add ed to open a particular map nam e. The session can be controlled from other w ind ow s via ovstop ovw and ovstart ovw com mand s 211 Fognet’s Field Guide to OpenView NNM There is a very hand y new er op tion to the ovw com m and that checks the health of XServer p eriod ically: ovw -pollXServer This op tion is also available as an OVw $APP_DEFS resou rce so it can be invoked au tom atically (see p age 9). On any virtu al w ind ow s server, m ake su re that the color d ep th is set for 24. If it is set low er, ovw m ay generate the follow ing error: “Cannot allocate 128 colors, Using m onochrom e im ages.” More abou t this error can be fou nd on p age 20. There are several tools that can be u sed to m ake su re the virtu al console stays locked . The follow ing site, for exam p le, has a tool for Wind ow s that locks ou t the keyboard and m ou se: w w w.e-m otional.com/ tscreenlock.htm An op en sou rce UN IX client to op en Wind ow s Term inal Server sessions can be fou nd at: w w w .rd esktop .org Web GUI login passw ord The d efau lt u sernam e/ p assw ord for w eb GUI login is ovu ser/ ovu ser. The p assw ord can be changed by ru nning: ovhtpasswd <user> Web GUI access using jovw registration files Jovw registration files can be u sed to control WEB GUI u ser access u sing stand ard htgrou p file d efinitions. In the htgrou p file, ad d grou p s w ith the u sers need ed . In the jovw registration file ad d sections like this: <110> "map1 IP Network" Access group1 Icon "launcher/network.16.gif" ActiveHelp { "><Access to the Network Presenter which displays the objects based on map1's Network."} f.action ―map1‖; Adding Web GUI menu and toolbar items To ad d a toolbar bu tton, m od ify the jovw files in both: $OV_WWW_REG/ Launcher/ C/ Jovw $OV_WWW_REG/ Jovw/ C/ Jovw Exam p le of a toolbar item to ping the selected d evice: 212 Web Interface ToolbarButton <100> @"C/toolbar/newwin.24.gif" Context "AllContexts" f.action "ping"; Action "ping" { MinSelected 1; URL "/OvCgi/webappmon.exe?app=IP+Tables&act=ping&help=on&sel=OV wSelections; WebWindow "ping" { Type limited; Width 600; Height 300; } } N ote the u se of webappmon, w hich is the Web GUI equ ivalent of xnmappmon (see p age 167). The cod e exam p le above ad d s the bu tton, bu t u ses the sam e GIF as another bu tton on the toolbar. Change the GIF by ad d ing the p ath to a 24x24 bit p ictu re. Menu bar item s for the Web GUI are sp read am ongst several files. For exam p le, the "p ing" m enu item in jovw is located in: $OV_WWW_REG/ jovw / C/ snm pview er Jovw m enu registration files u se the sam e syntax as stand ard N N M ARFs (see p age 6). Icons OK in ov w but not OK in w eb GUI Som e third p arty p rod u cts d evelop ed for N N M create cu stom bitm ap s for ovw (w hich su p p orts several grap hic form ats), bu t not in gif form at for the Web GUI. The gif form at files cau se qu estion m arks to ap p ear in the w eb GUI. The follow ing u tility can be u sed to convert the bitm ap s to gif form at: $OV_CON TRIB/ NN M/ ImageMagik/ convertBitmaps.ovpl N ote that on UN IX, $OV_CON TRIB/ N N M/ Im ageMagik shou ld be ad d ed to the LD_LIBRARY_PATH environm ent variable and convertBitm ap s.ovp l need s to be ru n as the root u ser. The new sym bols shou ld be d eceted right aw ay for jovw view s. To force Dynam ic View s to recom p u ted its sym bol m ap p ings after converting bitm ap s, ru n the ovdvstylereset com m and fou nd in $OV_BIN and then refresh any op en d ynam ic view s to ap p ly the new sym bols. 213 Fognet’s Field Guide to OpenView NNM javaGrapher contributed data collection tool javaGrap her is a contribu ted ap p lication and is a tool for live qu eries of the snmpCollect d ata. It d oes not, how ever, p ick u p u ser-configu red collections and is d esigned for m ore ad -hoc-typ e qu eries of SN MP objects. It also d oesn‟t seem to w ork w ith the base installation of N N M 7.5. It is located in: $OV_CON TRIB/ NN M/ javaGrapher Running JAVA apps in event action callbacks Use cau tion w hen calling JAVA ap p lications as au tom atic actions associated w ith N N M events becau se a sep arate JVM is started for each instance for each event lau nched and this can consu m e a lot of system overhead . That said , the typ ical p roblem one encou nters is that the classp ath is not set in the calling scrip t, for exam p le, the follow ing error is seen in ovactiond .log: Exception in thread "main" java.lang.NoClassDefFoundError: To resolve this p roblem , set the classp ath in the calling scrip t u sing: java –classpath <path_to_script> Secure JAVA-based telnet or ssh In som e environ m ents, GUI u sers d on‟t have the ability to locally access the d evices m anaged by N N M (jovw relies on this to lau nch local u tilities for telnet, etc). There are a few free JAVA -based telnet clients that fit nicely in this situ ation; here are links to tw o of them : http:/ / javassh.org/ http:/ / w w w.m ud .de/ se/ jta/ Setting the preferred w eb brow ser To configu re the brow ser u sed to lau nch URLs from w ithin N N M m enu s and ad d itional actions, see the ovw eb.conf m an p age and u se the $OV_CON F/ ovw eb.conf configu ration file. An exam p le entry (UN IX): Browser: /opt/ns-communicator/netscape -install %s Port: 3443 To reu se an op en N etscap e instance rather than op en a new one: Browser: /opt/netscape/netscape -remote "openUrl (%s,_blank)" || /opt/netscape/netscape %s 214 Web Interface Web GUI access control User control for Web GUI access This can be achieved by creating grou p s in the $OV_WWW/ etc/ htgrou p file w ith p articu lar u sers assigned to a grou p . Then, by assigning an access clau se for that grou p in the action block of a sp ecific action in the ap p lication's lau ncher registration file, the action can be restricted to the u sers of the grou p . The ap p lication's registration file for ovlau ncher is fou nd in the d irectory: $OV_WWW/ registration/ launcher/ C/ H ere is an exam p le $OV_WWW/ etc/ htgrou p file: NetworkAdmin: Mary NetworkOper: Rachel NTOper: + UNIXAdmin: Charlie UNIXOper: Jake OVAdmin: Charlie Jake Su: Molly Max Goldie NOC: Molly H ere is an exam p le jovw registration file snip p et to restrict the d isp lay of m ap op tions w ithin ovlau ncher to certain gr ou p s. /* * jovw.exe takes MapName and ObjectName parameters. * MapName specifies the map for the Network Presenter * to open, and ObjectName specifies the selection name * of an OVW object to display. */ Action "NNMNetworkPresenter" { URL "/OvCgi/jovw.exe?MapName=default"; WebWindow "OvWwwNetworkPresenter" { Type limited; } } Action "NNMNetworkPresenterIP" { URL "/OvCgi/jovw.exe?MapName=default&ObjectName=IP+Internet"; WebWindow "OvWwwNetworkPresenter" { Type limited; } Access NetworkAdmin; } /* The following two entries to the Action block are provided as examples of how to add personalized maps. You must also update the List blocks within this file. */ Action "NetAdmn" { URL "/OvCgi/jovw.exe?MapName=NetAdmin&ObjectName=IP+Internet"; WebWindow "OvWwwNetworkPresenter" { Type limited; } Access NetworkAdmin; } Action "NNMNOC" { 215 Fognet’s Field Guide to OpenView NNM URL "/OvCgi/jovw.exe?MapName=NOC&ObjectName=IP+Internet"; WebWindow "OvWwwNetworkPresenter" { Type limited; } Access NOC; } Action "NNMSu" { URL "/OvCgi/jovw.exe?MapName=Su&ObjectName=IP+Internet"; WebWindow "OvWwwNetworkPresenter" { Type limited; } Access Su; } Action "NNMNetworkPresenterHelp" { URL "/OvCgi/OvWebHelp.exe?Content=cntref&Context=cxtref&Scope=sc pref&Topic=Network_Presenter"; WebWindow "OvHelp" { Type intermediate; } } } Web GUI and ovw auth files Which rem ote system s that have access to the Web GUI can be controlled throu gh the $OV_CON F/ ovw .au th and $OV_CON F/ ovw d b.au th files. By d efau lt, all system s have access to the Web GUI. Both files m u st be u p d ated if u sing these files to set granu larity to sp ecific nod es. Alw ays m ake su re there is an entry for the N N M server itself in these files. 216 20. Extended Topology Extend ed Top ology (ET) is literally an entire p rod u ct w ithin the N N M p rod u ct. In 2000, the ET sou rce cod e w as licensed by H P from Riversoft Technologies (Riversoft w as acqu ired by Microm u se in 2002 and Microm u se w as acqu ired by IBM in 2005). H P has been integrating ET into N N M in stages ever since and ET has increasingly becom e the d om inant architectu ral com p onent of N N M in term s of u ser interface d evelop m ent and rep resentation of new top ologies. This has m anifested itself as Dynam ic View s in N N M (See section 22). In essence, Extend ed Top ology p rovid es visibility to Layer 2 connectivity, H SRP, VRRP, VLAN S, IPv6, and overlap p ing ad d ress sp aces that are otherw ise im p ossible w ith legacy N N M. When N N M is first installed , Extend ed Top ology is not enabled . The reason for this is that in highly-scaled environm ents, zones m u st be configu red or ET m ay becom e a p erform ance bu rd en. Also, a su ccessfu l ET d iscovery is d ep end ent on a w ell-d iscovered netmon-based top ology, and it is a good id ea to take a hard look at the resu lts of netmon d iscovery before enabling ET. Several red iscoveries of netmon-based top ology shou ld be exp ected d u rin g the im p lem entation p hase of m ost N N M d ep loym ents. Enabling Extended Topology Read the gu id e to Using Extend ed Top ology thorou ghly before enabling ET. When ru nning for the first tim e, u se: $OV_BIN/setupExtTopo.ovpl If this com m and is being ru n for the first tim e, it p rom p t s for zone configu ration. In environm ents w here there are few er than 200 connector d evices, selecting au tom atic zoning shou ld be OK. Zone configu ration can alw ays be u p d ated throu gh the Extend ed Top ology Configu ration w ind ow available from the Discovery Statu s tab in H om ebase. In general, zoning in sm aller N N M d ep loym ents w ill ad versely affect p erform ance. The best p ractice is to avoid u sing m ore than one zone. 217 Fognet’s Field Guide to OpenView NNM Dep end ing on the environm ent (and the version of N MM), d iscovery m ay take som e tim e. It can be m onitored in H om ebase u nd er the Discovery Statu s tab. ET autozoning Au tozoning is p rom p ted for by setu p ExtTop o.ovp l. The zones created by this featu re are strictly based on Layer 3 inform ation from the classic N N M top ology. All su bsequ ent top ology d iscovery ru ns w ill not u se the au tozone featu re and w ill then p lace any new ly -d iscovered d evices into the d efau lt zone. A best p ractice, then, is to m anu ally ru n the au tozone scrip t before ru nning a new ET d iscovery. By d efau lt, a new d iscovery is ru n w hen a threshold of new ly d iscovered nod es is reached , bu t the d iscovery can be ad ju sted in the H om eBase extend ed top ology configu ration w ind ow . The au tozoning featu re can be lau nched u sing the “Configu re zones au tom atically” op tion in the extend ed top ology configu ration w ind ow or by lau nching the follow ing com m and : $OV_BIN/ autozone.ovpl Monitor p rogress in the $OV_PRIV_LOG/ au tozone.log file. Restart ovas after ru nning au tozoning u sing ovstop/ovstart. Test the zones u sing the test zones op tion in the extend ed top ology configu ration w ind ow . If ru nning N N M on m ore p ow erfu l hard w are, the nu m ber of nod es that are p laced in a single zone can be increased by m od ifying the numManagedObjects p aram eter in the follow ing file, bu t H P recom m end s not u sing a valu e of m ore than 10,000: $OV_CON F/ nnm et/ recVals.conf Manual zoning tips If exp eriening trou bles w ith the au tozone featu re, consid er creating the zones m anu ally. Use these gu id elines for the zones: Create one zone for core netw ork d evices Create one or m ore zones for the central site‟s L2 connecting d evices and key com p onents 218 Extended Topology Create one or m ore zones for external sites L2 connecting d evices and key com p onents Use d evice nam ing stand ard s as w ild card s in the zone d efinitions. Su p p orted w ild card s in zone d efinitions are: * ? [..] [!..] n-n Any num ber of characters up to the next period (.) A single character A single character or a range of characters [acd ] or [a -f] Strings NOT containing a character or range [!acd ] or [!a -f] From -to for ranges of IP ad d resses Rem em ber to test all zones u sin g the test zones op tion in the extend ed top ology configu ration w ind ow . The ou tp u t files created by this op eration is $OV_TMP/ etzonetest.ou t and this file contains a list of all the d iscovered nod es and w hat zones they are p laced in. Restart or force ET discovery See p roced u re on p age 10 ET device agents Extend ed Top ology m akes d evice-sp ecific qu eries to su p p orted d evices to bu ild connectivity, top ology, and intelligence for the p olling engine. The follow ing URL‟ N N M d evice and protocol support: w ww .openview .hp.com/ prod ucts/ nnm et/ support/ d evice_support.htm l N N M d evice support: openview.hp.com/ prod ucts/ nnmet/ support/ d evice_requirem ents.htm l ET discovery and SN MP configuration SN MP com m u nity strings are tran sferred to N N M/ ET from the legacy SN MP configu ration su bsystem , bu t in V7.53 and p rior, SN MP p olling tim eou t and retry valu es are not transferred . In V7.53, tw o op tions are available for cu stom izing tim eou ts and retries in ET: the first is to im p ort the settings from legacy SN MP configu ration; the second it to configu re ET Discovery p olling tim eou ts and retries ind ep end ently. To enable im p orting of classic SN MP configu ration tim eou ts and retries, ru n: ovstop ovet_dhsnmp then ad d the –useNNMSNMPConf sw itch to the ovet_d hsnm p .lrf file p er the LRF p roced u re on p age 5, then restart the ovet_dhsnmp p rocess. 219 Fognet’s Field Guide to OpenView NNM For ET-based Discovery tim eou ts and retries, they are configu red in the m _Tim eOu t and m _N u m Retries p aram ters near the end of th e follow ing configu ration file: $OV_CON F/ nnm et/ DiscoSnm pH elperSchem a.cfg. For changes to take effect, restart the ET p rocesses u sing etrestart.ovp l. The below logfile w ill ind icate if tim eou ts are occu rring d u ring ET d iscovery: $OV_PRIV_LOG/ ovet_d hsnm p.log ET discovery considerations and limitations ET m akes u se of d ata in the forw ard ing (FDB) tables of LAN Sw itches w hen com p u ting top ological relationship s. Many sw itches d o not generate layer-2 p ackets that contain their ow n layer -2 ad d resses. This resu lts in the forw ard ing tables of neighboring sw itches not having entries for each other. This resu lts in too m any sw itch-to-sw itch connections to be created by ET w hen sw itches are not d irectly connected to each other. ET can take ad vantage of p rotocols like CDP to elim inate non-neighbor connections in the top ology. Su ch p rotocols "ad vertise" the p hysical ad d resses via layer-2 p ackets. This resu lts in u sefu l connectivity inform ation being available in SN MP forw ard ing tables. Also, w hen VLAN s are in u se in an environm ent (and d iscoverable via su p p o rted vend or-sp ecific SN MP agents), "non-neighbor connections" can often be recognized as su ch. Below are ad d itional consid erations regard ing sw itches: 220 Exp ect H P to d evelop im p rovem ents to the "top ology stitching" algorithm s in fu tu re releases to N N M, keep u p -to-d ate on N N M p atches and on ET d evice agents. With Cisco d evices, CDC shou ld be u nd erstood and cu stom ized for the environm ent. See p age 180 for m ore inform ation on CDC. ATM sw itches that su p p ort the ILMI MIB p rod u ce m ore accu rate and com p lete ATM sw itch connectivity . The ILMI d iscovery agent only d iscovers rem ote neighbors for interfaces w ith ifTyp e of atm (37). Devices su p p orting the ATM MIB that rep ort ILMI neighbors for other interface typ es, for exam p le sonet(39) w ill not be d iscovered . Extrem e N etw orks d evices ru nning OS version 6.1.7b7 or later, enable SN MP access to the forw ard ing d atabase: Extended Topology enable snmp dot1dTpFdbTable ET d iscovery m ay treat certain rou ters as "end -nod es" u nd er circu m stances w here there is not enou gh d ata to create rou ter-to-rou ter connections. ET in N N M V7.53 and p rior d oes not have su p p ort for the Rep eater MIB (rfc2018) or for enterp rise-sp ecific rep eater and / or hu b MIBs. N N M, how ever, continu es to interrogate d evices su p p orting rfc2018, as w ell as the H P Interconnect Fu nction ality (ICF) MIBs, w hich su p p ly connectivity inform ation for N eighbor and N od e View s. Improving ET discovery accuracy The follow ing step s m ay im p rove the accu racy of the top ology d iscovered by ET: 1. 2. 3. 4. Enable d iscovery protocol (CDP, FDP, EDP) to im prove accuracy If the above protocols aren‟t available, run d iscovery d uring an active tim e on the netw ork to better grab Forward ing Database tables in sw itches Consid er the need to m anage end -nod es. Managing end -nod es im proves L2 connectivity accuracy but may cause more events Check DN S perform ance. ET uses DN S lookups extensively The CDC introd u ced in V7.51 (See Page 180) m ay affect the accu racy of d iscovered d ata. The CDC is not enabled by d efau lt how ever. Improving ET discovery performance The p erform ance of ET d iscovery and the APA p oller has been d ram atically im p roved throu gh several version s of N N M and p articu larly throu gh increm ental p atches to V7.5. When encou ntering issu es w ith p erform ance alw ays m ake su re the latest version and the latest p atches are installed . As w ith netmon, ET d iscovery p erform ance is also d irectly affected by DN S p erform ance. See Section 4 on DN S for m ore on this. N N M V7.51 introd u ced CDC to p rovid e ad m inistrators a w ay to control how m u ch d ata is p u lled from Cisco d evices d u ring ET d iscovery. This featu re can greatly im p rove the p erform ance of ET d iscovery, bu t d oes so at the p otential exp ense of accu rate top ology rep resentations. See Page 180 for d etails on the CDC. Interface filtering, w hich w as introd u ced in a p atch to V7.5 can greatly sp eed ET d iscovery in highly-sw itched environm ents. More 221 Fognet’s Field Guide to OpenView NNM inform ation on this featu re can be fou nd in the InterfaceFiltering w hite p ap er and on the section on the netm on.interfaceN oDiscover file on Page 53. Dynam ic View s by d efau lt u ses a com bination of filters from ET filter files (p aConfig.xm l and Top oFilters.xm l) and the legacy N N M "filters" file ($OV_CON F/ C/ filters). Disabling the legacy filters file sim p lifies filters d efinitions and boosts p erform ance. If ru nning N N M versions p reviou s to N N M 7.53, change the "enableETFilters" p aram eter from 0 to 1 in: $OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l If ru nning N N M 7.53, change the "enableETFilters" p aram eter from 0 to 1 in: $OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ 1Nod eServlet.xm l Reducing ovet_poll startup time If startu p tim e for ovet_p oll is taking m ore than a m inu te or tw o, there m ay be a large nu m ber of u nconnected interfaces in the environm ent. The follow ing setting m ay red u ce startu p tim e: Ed it $OV_CON F/ nnm et/ top ology/ filter/ Top oFilters.xm l Find the section below and com m ent it ou t: <!-<filter name="NotConnectedIF" objectType="Interface" title="NotConnectedInterfaces" description="Interfaces without Layer 2 Connection"> <operator oper="NOT"> <filterName>ConnectedIF</filterName> </operator> </filter> --> It is com m ented by ad d ing <!-- as com m ent start, and --> at the end . Then, locate the follow ing line: <interfaceAssertion name="ConnectedIF" title="ConnectedInterfaces" description="Interface with L2 Connection"> 222 Extended Topology And ad d this section above it: <interfaceAssertion name="ConnectedIF" title="NotConnectedInterfaces" description="Interface without L2 Connection"> <operator oper="NOT"> <attribute> <capability>isL2Connected</capability> </attribute> </operator> </interfaceAssertion> Check filter validity with ovet_topodump.ovpl –lfilt, the restart ovet_poll. ET and ov w database synchronization issues Trou ble can resu lt w hen objects are d iscovered by netmon, bu t have not yet been d iscovered by ET. Prior to V7.51, there is no single com m and to force a single object or set of objects to be d iscovered by ET. To force a com p lete ET d iscovery, or a single zone d iscovery, see the p roced u re on p age 10. To synchronize a nod e that has not been d iscovered by ET bu t has been d iscovered by netmon in V7.51, see the p roced u re im m ed iately below . ET single/incremental node discovery After V7.51, single nod es that are su bsequ ently ad d ed by netm on can be m ad e a p art of Extend ed Top ology, bu t there are som e lim its in the initial su p p ort for this featu re. For exam p le, connectivity for new ly ad d ed nod es is not yet su p p orted . It is im p ortant to read the Increm entalN od eDiscovery.p d f w hite p ap er for m ore inform ation on u sing this featu re. In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. This featu re is d isabled by d efau lt. To enable this featu re, ad d the follow ing LRF flag to the ovet_brid ge.lrf file and u sing the LRF p roced u re on Page 5: -single_nod e_d iscovery Restart the ovet_bridge p rocess after u p d ating the LRF u sing ovstop/ovstart. Prior to V7.53 Interm ed iate Patch 20, Single N od e Discovery d iscovers interfaces on non SN MP nod es and m arks them as d isabled . 223 Fognet’s Field Guide to OpenView NNM Since the interfaces are u nconnected , p olling is controlled by a p aConfig.xm l filter that u ses the new isN ew N od e and isN ew Interface attribu tes on the new ly ad d ed nod es. These attribu tes are cleared after a fu ll or increm ental zone d iscovery occu rs. To initiate a nod e d iscovery after enabling this featu re, sim p ly issu e a stand ard d em and p oll to the nod e u sing ovw m ap fu nction or from the com m and line u sing nmdemandpoll. Increm ental nod e d iscovery activities are logged to: $OV_PRIV_LOG/ ovet_brid ge.log D etect interface configuration changes The below configu ration settings shou ld be consid ered if the APA is enabled . Ad d ed in N N M V7.53, the follow ing LRF flag ad d ed to the ovet_brid ge.lrf file (u sing the LRF p roced u re on Page 5) w ill enable au tom atic d iscovery of interface config u ration changes: -interface_config_change Sim ilarly, the follow ing LRF flag w ill au tom atically d etect SN MP interface ind ex table renu m bering: -hand le_interface_renum ber N ote that au tom atic d iscovery of interface changes and interface renu m bering settings are d isabled by d efau lt. Restart the ovet_bridge p rocess after u p d ating the LRF u sing ovstop/ovstart. Detection of the above changes occu rs throu gh the APA‟s m onitoring of the sysUpTime and entLastChangeTime MIB variables. By d efau lt, the form er is en abled by d efau lt, bu t the later isn‟t. To enable entLastChangeTim e m onitoring, ed it the p aConfig.xm l file and set the entityMibEnable p aram eter in the ConfigPollSettings to “tru e.” H P recom m end s that if the above p aram eter is set to tru e, a new filter is created to id entify d evices that su p p orts the entLastChangeTime MIB variable and then setting u p the ConfigPollSettings so that the entLastChangeTime p aram eter is set to “tru e” only for su p p orted d evices. Also, lim itations ap p ly to the classes of interfaces that the above settings ap p ly to. The d ocu m entation for this m ay be fou nd in the Gu id e to Using Extend ed Top ology u ser m anu al. 224 Extended Topology D etect interface table additions N N M V7.53 introd u ced the ifNumberQueryEnable p aConfig.xm l file p aram eter setting in the config grou p w hich w hen set to tru e (false by d efau lt), w ill cau se the APA to p oll the ifN u m ber variable w ith every APA statu s p oll. If ifN u m ber increm ents, the APA w ill generate the new event OV_APA_IF_ADDED. Zone discovery tips and performance tricks Zones are an ET scalability featu re and an absolu te necessity for environm ents that m anage m ore than a few hu nd red connector d evices. As a general ru le, zones shou ld corresp ond to geograp hic locations or other logical grou p ings w here the nu m ber of links betw een the grou p s is m inim ized . Do not sep arate d irectly-connected sw itches, sw itches connected to rou ters, or sw itches in the sam e global VLAN . Zones shou ld be connected by having at least one rou ter from each zone in another zone. setupExtTopo.ovpl au tom atically p artition s d evices into d iscovery zones. It su ggests zones that m aintain sw itch and rou ter connection, m inim ize overlap and d u p licate SN MP qu eries, and im p rove p erform ance. Zone configu ration occu rs after ET d iscovery, and is based on the assu m p tion that the d iscovery w as com p lete and accu rate. Manu al m anip u lation of the the zone of a p articu lar d evice can be accom p lished by ed iting the follow ing file then restarting ET: $OV_DB/ nnm et/ hosts.nnm N ote that “0” is the d efau lt zone and the file is overw ritten w henever ET d iscovery takes p lace (ed iting this file is not su p p orted by H P). The sizing of the zones is configu red w hen enabling ET w ith setu p ExtTop o.ovp l. The valu es u sed are stored in: $OV_CON F/ nnm et/ recVals.conf. The valu es u sed are based on the available RAM, sw ap sp ace, and the nu m ber and sp eed of CPU's in the system . The valu es can be cha nged after ru nning setu p ExtTop o.ovp l, bu t before ru nning Au tozone. N ote changed valu es w ill be overw ritten w hen re-ru nning setu p ExtTop o.ovp l. The valu es w hich give m ost p erform ance im p rovem ents are: nu m Managed N od es and nu m Managed Object. Be 225 Fognet’s Field Guide to OpenView NNM carefu l increasing them too m u ch, how ever, since lack of accu racy in the top ology can resu lt. Passw ord to access ET Configuration page setupExtTopo.ovpl p rom p ts for a u sernam e and p assw ord w hen ru n. This info is stored in ad m inistrator block in the follow ing xm l file u nd er the m ain N N M installation d irectory: $OV_AS\webapps\topology\WEB-INF\dynamicViewsUsers.xml Connection editor Introd u ced in V7.01, the conn ection ed itior allow s d irect m anip u lation of the ET top ology d ata. H P ad d ed som e d ocu m entation on the Connection Ed itor featu re in V7.51 w hich can be fou nd in : $OV_DOC/ WhitePapers/ ConnectionEd itor.pd f In V7.53, this d ocu m entation is rolled u p in Chap ter 2 of the Gu id e to Using Extend ed Top ology u ser m anu al. The d ocu m entation d iscu sses a m ore generalized ap p roach to connection ed its that can be u sed to sp an nod es in sep arate zones and u ses the follow ing file: $OV_DB/ nnm et/ generalConnsEdits Prior to v7.51, H P p rovid ed m ore a bit m ore lim ited connection ed it fu nctionality throu gh the follw ing file: $OV_DB/ nnm et/ connectionEd its Only end p oints that alread y exist in top ology d ataba se can be connected throu gh both files. The files contain OQL qu eries that p erform the d esired op erations, for exam p le: To ad d a connection betw een nod es Bobby and Jerry, both u sing ifIndex 91: insert into disco.connectionEdits (m_Name, m_NbrName, m_Command) values ('bobby.dead.net [ 0 [ 91 ] ]', 'jerry.dead.net[ 0 [ 91 ] ]', 1); To rem ove an invalid conn ection betw een nod es Bobby and Jerry, both u sing ifInd ex 91: 226 Extended Topology insert into disco.connectionEdits (m_Name, m_NbrName, m_Command) values ('bobby.dead.net[ 0 [ 91 ] ]', 'jerry.dead.net[ 0 [ 91 ] ]', 2); The last d igit in the target statem ent is the com m and and is one of the follow ing: 1 2 0 Add connection Delete connection Do nothing To u se the connection ed itor, Create, then ed it: $OV_DB/nnmet/connectionEdits Once the above file is created , ru n: ovet_topoconnedit.ovpl Discovery shou ld not have to be re-initiated , bu t if the exp ected u p d ates d o not occu r, force an ET d iscovery p er p roced u re on p age 10. To d eterm ine the p rop er ifInd ex to sp ecify, u se: $OV_SUPPORT/NM/ovet_topoquery getNodeByName <target> To d u m p all the level 2 connection in the N N M/ ET top ology: ovet_topoconndump.ovpl End nodes not in same zone as connector To d iscover connections for nod es that end u p in d ifferent zones from their corresp ond ing connector d evices, m od ify the DiscoSchem a.cfg file p er the instru ctions in the section on Connectivity in Chap ter 2 of the Gu id e to Using Extend ed Top ology u ser m anu al. Add connections for unsupported devices ET top ology m ay not be com p lete, or, in som e cases, incorrect w hen a d evice that is not su p p orted by the d evice agents is in p la y. In general, the connection ed itor can be u sed to d elete inaccu rate connections that m ay be assu m ed by ET from the d evice and ad d new connections to p rop erly show the correct connections. An excellent exam p le is p rovid ed in Chap ter 2 the Gu id e to Using Extend ed Top ology u ser m anu al. 227 Fognet’s Field Guide to OpenView NNM Layer 3 edge connectivity Ed ge connectity is assu m ed by ET w hen a su bnet is shared by only tw o ad d ressed interfaces. This assu m p tion m ay lead to erroneou s layer 3 connections in the ET top ology. To stop ET from creating connections based on this assu m p tion, u ncom m ent the line that read s: #enableConnectivity:1 and change “1” to “0” so the line read s: enableConnectivity:0 in the follow ing file: $OV_CON F/ nnm et/ Ed geL3Conn.cfg Ru n a new ET d iscovery cycle p er the p roced u re on p age 217. Exclude nodes from ET discovery The brid ge.noDiscover file can be u sed to exclu d e netmon-d iscovered d evices from being p laced in the ET top ology. Also, w ith the release of N N M 6.31, H P introd u ced the ability of filters to be u sed to p revent d iscovery of w hole classes of nod es based on stand ard N N M filters. For m ore inform ation on Discovery Filters, see Page 39. Show ing device details for non-SN MP devices Inform ation for nod es that ET can‟t otherw ise d iscover can be im p orted into the ET d atabase for som e lim ited field s inclu d ing SysN am e, SysContact, and u ser d efinable field s. The ETN onSnm p N od eDataIm p ort.p d f w hite p ap er (or, if ru nning V7.53, the Gu id e to Using Extend ed Top ology u ser m anu al.) d escribes this featu re, w hich requ ires 10 exp licit field s and is configu red via the follow ing configu ration file: $OV_CONF/nnmet/nonsnmpnodes.nnm. Use the follow ing com m and to check the syntax of the configu ration file after m aking changes to it: $OV_SUPPORT/NM/testnonSnmpFile.ovpl nonsnmpnodes.nnm Cu stom Icons/ Sym bols can be assigned to non -SN MP d evices by u sing the follow ing sp ecial sysObjectID: 1.3.6.1.4.1.11.2.3.16.n ”n” is the OID instance assigned in the nonsnm p nod es.nnm that corresp ond s to a sp ecific entry in that file. 228 Extended Topology Sw itching routers and routing sw itches in ET ET issu es cu stom SN MP p olls to d evices that are then u sed to convey top ology, connectivity and statu s inform ation. Since m any d evices in this class p rovid e m u ltip le m anagem ent card s, it is im p ortant that they are configu red to best reflect the actu al u se of the d evice and its relationship to other d evices in the netw ork. In essence, if the d evice is set u p to fu nction m ore like a sw itch than a rou ter, the m anagem ent card for sw itch fu nctions shou ld be enabled , and vice versa if the d evice acts m ore in a layer 3 rhelm . If the d evice tru ly p rovid es both fu nctions, it m ay be p ossible, u sing DN S, to ad d ress both m anagem ent card s as sep arate entities w hich w ill show u p in the top ology as sep arate d evices. See Page 88 for m ore on ET interaction w ith d evices of this typ e from an APA p olling p ersp ective and Page 205 for m ore on netmon d iscovery consid erations and legacy ovw im p lications. ET command summary, support tools, log files The follow ing com m and s are u sed to create and m anip u late the Extend ed Top ology: setupExtTopo.ovpl autozone.ovpl etrestart.ovpl –verbose –disco ovet_topoconnedit.ovpl Enable Extended Topology Run autozoning Force ET discovery ET connection editor The follow ing log files can be u sed to trou bleshoot issu es w ith ET: $OV_PRIV_LOG/ovet_bridge.log $OV_LOG/ovas.log $OV_PRIV_LOG/ovet_disco.log Bridge logfile Tomcat server logfile ET Discovery logfile The su p p ort com m and s listed below can all be fou nd in the follow ing d irectory: $OV_SUPPORT/ N M $OV_BIN/ovet_topodump.ovpl There is good docum entation in the m an/ ref pages for this com mand w hich d um ps ET topology data. ovet_demandpoll.ovpl Poll nodes, update status bridge,, enable/disable polling, and tracing. Note the -d (d um p) param eter changed in patch 15 to NN M V7.51 and gives d etailed info on w hich filters are in use for interfaces and nod es. More on page 72. checkPollCfg 229 Fognet’s Field Guide to OpenView NNM This tool displays the APA polling settings for a node, its interfaces, addresses and boards that are configured in paConfig.xml. Note that the displayed values only reflect settings in the paConfig.xml file, not the runtime ovet_poll process. ovet_topoquery Use this versatile tool to d um p detailed d ata from the ET database. Also, this tool can be used to change the status of ET topology objects w ith the setNodeStatus and setIfState flags. Run this tool with the und ocum ented –internal flag to d um p an extensive list of flags for this com mand. ovet_fixTopology.ovpl This tool helps fix anom alies th at are attributable to SN MP tim eouts d uring d iscovery. The script can be run w ith the –checkDanglingConn option to see if the fix option is need ed. ovet_generateTopoDeltaReport.ovpl This tool com pares the current d iscovery pass w ith the last one. ovet_ toponame2id.ovpl This tool converts a ovw selection nam e to the ET object ID ovet_ toposet.ovpl This tool sets or unsets polling for a object, see page 75 ovet_reloadTopoDBTbls.ovpl This tool d rops and recreates all N N M Extend ed Topology tables w ithout d ropping the NN M data warehouse tables. ovet_topoobjcount.ovpl This tool outputs totals for d ifferent kind s of objects in the ET d atabase if called w ith the –all option. ovtopodbsnapshot.ovpl This tool m akes a snapshot of the ET topology data . dumpDiscoStatus.ovpl Displays ET d iscovery status. All configured agents m ust get to state 4. Valid state values are: 0: Und efined 1: Not Running 2: StartUp 3: Running 4: Finished 5: Died dumpAgentProgress.ovpl Displays ET d iscovery status for an ET agent. ETsNoSnmpNodes.ovpl 230 Extended Topology This tool outputs the IP ad d resses of nod es that d id no t respond to Extend ed Topology setup SN MP query and can help id entify unresponsive SN MP nod es. ovet_truncatetopotbls_all.ovpl This script is used to truncate all Solid tables used by the topology service. It can be used to determ ine if there are interfaces that are m isconfigured in ET. misconfigIfaces.ovpl This script find s interfaces w ith speed and / or d uplex m ism atch . zoneTime.ovpl This tool gives info about the tim e used for d iscovery d uring the last ET d iscovery.. ET and D N S issues Errors sim ilar to the follow ing m ay ap p ear in $OV_LOG/ ovet_au th.log: 2002-05-15 17:07:48 rvd: unrecoverable IP configuration error: gethostname() returns '<hostname>', gethostbyname() for that returns IP address '15.2.113.5', but that address does not match any listed interface. 2002-05-15 17:08:07 rvd: startup aborted: Initialization failed. Fatal warning : A critical bus error occurred. in ../CRivRvNet.cc at line 367 These errors incicate DN S configu ration issu es and ovet_au th m ay d u m p core and exit. Check the forw ard and reverse DN S configu ration for the affected nod e. 231 21. Dynamic Views Dynam ic view s are available throu gh the H om ebase GUI, w hich p rovid es Extend ed Top ology rep resentations. Dynam ic view s are not available u nless Extend ed Top ology is enabled , w hich it is not w hen N N M is initially installed . It is im p ortant to have a w ell-d iscovered netw ork top ology before enabling ET. Dynam ic View s are controlled by the ovas d aem on, w hich is registered throu gh the LRF p rocess. See p age 5 for tip s on controlling LRF d aem ons. H om ebase is able to p rop erly d isp lay the fam iliar level 3 rep resentations typ ical of the legacy ovw or Web GUI, neither of w hich is architectu rally able to su p p ort ET view s. H om eb ase is com p letely ind ep end ent of ovw and the legacy d atabases that su p p ort it. ET still relies on netmon for netw ork d iscovery (excep t for OAD d iscovery). From netmon d iscovery d ata, only basic object d ata is conveyed to the extend ed top ology d atabase, w hich ET u ses to again p oll the d evices for ET configu ration once ET is enabled . ET p robes d evices sp ecific agents for in -d ep th top ology d ata and H P m aintains a sep arate su p p ort m atrix for extend ed top ology d evice agents. Dynam ic View s has both server (ovas) and client (w eb p age w ith Java Plu g-in) com p onents. Dynam ic View s featu res inclu d e: Ad d and Delete nod es Expand all or connecting neighbors, path Poster printing: large view s can be printed in chunks Toggle port labels Tom cat-based au thentication D ynamic view s available Dynam ic view s available w ith N N M 7.5x. N ote that som e of these view s requ ire ad d itional licenses. H SRP, VRRP, OSPF, and IPv6 view s 232 Dynamic Views requ ire p u rchase of the Ad vanced Rou ting SPI. They are enabled by d efau lt w hen N N M Evalu ation cod e is installed , bu t exp ire along w ith the eval key. H om e Base Container View ET Discovery Status Alarm View (APA only)(T) N eighbor View Internet View Segm ent View Problem Diagnosis View CDP View IPV6 N etwork View IPV6 Interface Detail (T) Overlapping Add ress Domain View Interface View (T) (T) indicates table view N ode View Access Path View Path View N ode/ If Status(APA only)(T) Station View N etwork View VLAN View OSPF View H SRP View (T) IPV6 N od e Detail (T) IPV6 Prefix Groups (T) Container View VRRP View (T) V7.51 ad d ed container view s (Page 235) and access p ath view (Page 168). Several other view s can be ad d ed if u sing one or m ore of the ad d -on SPI's for N N M. V7.53 ad d ed the Alarm View , N od e Statu s View and Interface Statu s View . All three of these new view s are only available in the APA is enabled . D ynamic View s URLs Dynam ic View s URLS changed betw een N N M V6.31 and N N M V6.41. This is w hen the Tom cat server rep laced the ovp l/ ov-cgi-based d ynam ic view s. The form er and new er locations are resp ectively: http:/ / <NN M-Server>:3443/ OvCgi/ ovlaunch.exe http:/ / <NN M-Server>:7510/ topology/ Sp ecific view s can be accessed by URL d irectly, for exam p le: http:/ http:/ http:/ http:/ http:/ / / / / / <NN M-Server>:3443/ <NN M-Server>:7510/ <NN M-Server>:7510/ <NN M-Server>:7510/ <NN M-Server>:7510/ OvCgi/ nod eView.ovpl topology/ nod eView topology/ segm entView topology/ ospfView topology/ vlanView 233 Fognet’s Field Guide to OpenView NNM D ynamic View s configuration The follow ing tw o files are the m aster configu ration entry p oints for Dynam ic View s: $OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l $OV_CON F/ d ynam icView s.conf With the release of N N M 7.53, m any configu ration p aram ters w ere sp lit off from w eb.xm l and p laced in context-sp ecific xm l files in: $OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ To m erge xm l files in the servletRegistration d irect ory w ith w eb.xm l, ru n the follow ing com m and : $OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl D ynamic View s concurrent view s/users The nu m ber of concu rrent u sers that can access Dynam ic View s is not lim ited to the actu al nu m ber of u sers, rather to the nu m ber of d ynam ic view s op en at any given tim e. acceptCount is the p aram eter w hich d efines this m axim u m . Change the d efau lt by m od ifying the acceptCount valu e for the connector classnam e for p ort 7510 in the $OV_AS/ server.xm l file and then ovstop/ovstart the ovas d aem on: <Connector className="org.apache.catalina.connector.http.HttpConnector" port="7510" minProcessors="5" maxProcessors="125" enableLookups="true" redirectPort="8443" acceptCount="10" debug="1" connectionTimeout="-1"/> D ynamic View s access via Webstart On Wind ow s clients, JAVA Webstart can be u sed to access the Dynam ic View GUI w ithou t u sing a stand ard w eb -brow ser. This is d ocu m ented in the follow ing u ser m anu al: N etw ork N od e Manager v7.5 Dynam ic View s ap p lication integration w ith Java Web Start Ap p lication Manager. Webstart can be installed from : http:/ / nnm -server:7510/ topology/ w ebstart Problem s m ay arise in accessing w ebstart d u e to the fact that H P p op u lates the .jnlp file w ith the short nam e for the server. If an “Unable to load resou rce” error m essage is d isp layed , find the three lines w ith the N N M server nam e in the follow ing file and change them to u se the FQDN : 234 Dynamic Views $OV_AS/ w ebapps/ topology/ w ebstart/ d vclient.jnlp D ynamic view s access control The follow ing scrip t p rovid es the front-end for m anaging u sers w ho have access to d ynam ic view s: $OV_BIN/ d vUsersManager.ovpl By d efau lt, d ynam ic view s are op en to all, excep t for: Configuring Extend ed Topology Managing/ unm anaging nod es Ad d ing/ d eleting nod es To ad d a u ser role for an op erator or an ad m inistrator via Tom cat realm s if ru nning versions of N N M p reviou s to 7.53, u ncom m ent the block sim ilar to below in $OV_AS/ w ebap p s/ top ology/ WEBIN F/ w eb.xm l: <security-constraint> <web-resource-collection> <web-resource-name>DV Access Secure</web-resource-name> <url-pattern>/*</url-pattern> </web-resource-collection> <auth-constraint> <role-name>operator</role-name> <role-name>administrator</role-name> </auth-constraint> </security-constraint> If ru nning N N M 7.53, u p d ate the sim ilar block in the follow ing file: $OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ userAuthentication.xm l Then ru n the follow ing scrip t to ad d you r changes in the u serAu thentication.xm l file to the w eb.xm l file: $OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl Then ad d the u sers and assign roles in: $OV_AS/ w ebapps/ topology/ WEB-IN F/ d ynam icView sUsers.xm l <tomcat-users> <user name=―ovuser1‖ password=―mypw‖ roles=―operator‖/> <user name=―ovadmin‖ password=―adminpw‖ roles=―administrator‖/> </tomcat-users> 235 Fognet’s Field Guide to OpenView NNM After any of the above changes restart ovas via: ovstop ovas; ovstart ovas The above au theticalion m ethod is BASIC (cleartext p assw ord s are sent over the netw ork). To u se MD5 p assw ord s instead , enter the resu lts of the follow ing com m and as the p assw ord above, w here <p assw d > below is the p assw ord to be encryp ted : ―$OV_JRE"/bin/java -classpath \ "$OV_AS"/server/lib/catalina.jar:"$OV_AS"/bin/bootstrap.jar \ org.apache.catalina.realm.RealmBase -a MD5 <paawd> N ote on IPF, the p ath for the com m and is: $OV_JRE"/bin/IA64N/java Make su re LD_LIBRARY_PATH is set to: $OV_JRE/lib/IA64N:$OV_JRE/lib/IA64N/server N ext, ed it the $OV_AS/ conf/ server.xm l file and ad d the d igest p aram eter of the <Realm > elem ent of the top ology Context to d igest="MD5”, e.g: <Context path="/ topology" docBase="topology" d ebug="0"> <Realm classN am e="org.apache.catalina.realm .Mem oryRealm " pathnam e="w ebapps/ topology/ WEB-IN F/ d ynam icView sUsers.xm l" d igest="MD5" /> </ Context> D ynamic view s via SSL The follow ing Manu al w as ship p ed w ith N N M V7.53 to exp lain how to enable SSL for lau nching Dynam ic View s throu gh Java WebStart: $OV_WWW/ htdocs/ C/ manuals/ Configuring_SSL_for_Dynam icView s.pd f Container view s V7.5.1 introd u ced this p ow erfu l view set to rep lace the trad itional role of ovw view s in rep resenting cu stom ized view s of the netw ork w ith backgrou nd grap hics, etc. A w hite p ap er d escribing this featu re can be fou nd in: $OV_DOC/ WhitePapers/ ContainerAd m inistration.pd f In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. Unlike ovw-based containers, ET containers are not lim ited to the strict hierarchical constraints of 236 Dynamic Views IPMAP. In fact, containers can even be contru cted u sing circu lar references. Containers can be d efined d ynam ically u sing filters, w hich are evalu ated each tim e the container is op ened . APA statu s is p rop agated throu gh containers. The only real restriction on containers is that they m u st be hierarchical and each container nam e m u st be u niqu e. The Container View s m enu s p rovid e a set of container creation and cu stom ization op tions, w hich are w ritten to an xm l file: $OV_DB/ nnm et/ containers.xm l This file shou ld not be m anip u lated d irectly to create containers. Use the follow ing p roced u re to m anu ally u p d ate the containers.xm ll file: Backup the existing containers.xml file Create a working copy of containers.xm l and ed it as d esired overw rite the runtim e copy in $OV_DB/ nnm et w ith the working copy Run ovstop ovas ; ovstart ovas Reload the view s using the follow ing url option: http:/ / <hostnam e>:7510/ topology/ home?reloadContainers=true The w hite p ap er m entioned above contains an exam p le .xm l file and d escribes the variou s tags allow ed in this file. There are also tip s on trou bleshooting issu es w ith container view s. Container View customization best practices Ad d ing nod es d irectly to the cointainers.xm l is alw ays p referable to exp and ing a p articu lar container‟s view u sing GUI op eration and then u sing the “Save Layou t” op tion in the Container Setu p d rop d ow n m enu . The reason for this is that any objects w hich are ad d ed to the containers via the “Exp and ” op tions are only saved by object ID in t he containers.xm l file. This m akes ongoing m anagem ent u sing the xm l file d ifficu lt. Avoid u se of the “Inclu d e N od e . . .” m enu item w hen u sing filters. The tw o d on‟t m ix w ell. It‟s best to create filters and then strictly u se filters for containm ent. The “Inclu d e N od e. . .” m enu is really best u sed to test ou t a container view . For d ep loym ent, try to stick w ith filter-based containers. Jp eg (.jp g) files for backgrou nd im ages resize better than other im age files typ es. It is best not to show connectivity betw een containers by 237 Fognet’s Field Guide to OpenView NNM setting show ContainerConnectivity=“false” on all containers. There is an exam p le in the w hitep ap er. Don‟t p u t too m any nod es in a container. They are recalcu la ted every tim e they are op ened . Try to cap the size arou nd 200 nod es or so. If connectivity is not im p ortant to show in a container , set the below flag: show N od eConnectivity=“false” This m akes the view ‟s p erform ance im p rove greatly, so consid er setting this for all containers. A neighbor view for connectivity from a nod e can alw ays be lau nched to show connectivity . Connectivity w ithin a container can be p roblem atic becau se key nod es m ay not p ass the filter and m ay not be in the container, thu s p ossibly d egrad ing the accu racy of the connectivity. Make a second backu p of the containers.xm l file and call it containers.raw .xm l. Ed it this file and rem ove all references to object IDs. This w ill be u sefu l if the ET d atabase has to be w ip ed and re created from scratch. Container View operations If ru nning versions p rior to V7.53, th e below op erations are not covered in: $OV_DOC/ WhitePapers/ ContainerAd m inistration .pd f. In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. Delete a container: Ed it the file: $OV_DB/ nnm et/ containers.xm l. ovstop ovas w hen ed iting this file since it cou ld be changed by ovas. Rem ove any containerReference tags and the entire container tag for the container to be d eleted . Renam e a container: ovstop ovas and ed it $OV_DB/ nnm et/ containers.xm l and sim p le rep lace the nam e in the container nam e tag and in the containerReference nam e tag. 238 Dynamic Views Rem ove selected objects from a container: Objects that had been saved from the GUI d o not have nod e nam es associated w ith them in the contain er.xm l file. To rem ove objects of this typ e, m ove the objects to be d eleted into a corner of the screen, then “Save Layou t.” Ed it the containers.xm l file and the d esired objects can be id entified by their X and Y coord inates. Scaling container icons: To change the size of the container relativ e to the backgrou nd grap hic, the size of the backgrou nd grap hic grap hic m u st be increased . This can be d one in any grap hics ed iting tool su ch as Microsoft Pictu re Manager. Rep lace the im age files in: $OV_WWW/ htd ocs/ im ages/ backgrou nd s. Restart ovas or call the follow ing URL to reload the new im age: http:/ / <hostnam e>:7510/ topology/ hom e?reload Containers=true Container View s access control Container View s access control is only available in N N M V7.53 and later. By d efau lt, every container is accessible to every u ser. An access list can inclu d e a list of u sers and / or a list of roles. User -based access lists p rovid e a w ay to grant access to ind ivid u al u sers. Users w ith the role of ad m inistrator au tom atically have access to all containers, even if the container has an access list. An em p ty access control list allow s only ad m inistrators to access the container. Users and roles are d efined as a p art of the u ser au thentication p roces s. Before enabling container access control, enabling u ser au thentication is requ ired . See p age 235 for m ore inform ation on Dynam ic View s access control. To enable access control for a container, set the enableAccessControl to “tru e” or “false” in the container d efinition in the containers.xm l file. Then, ad d an access list sim ilar to the follow ing exam p le: <accessList> <user name="myuser"/> <role name="myrole"/> 239 Fognet’s Field Guide to OpenView NNM </accessList> If u ser access is configu red for containers, the access assignm ents ap p ly to the N od e Statu s View , Interface Statu s View , and Alarm View . In these view s, the nod es, interfaces, or alarm s a u ser can see are d eterm ined by the total set of nod es in all the containers to w hich the u ser has access. 240 Dynamic Views Container View example The exam p le below show s a container view that is p artially containerized . That is, som e of the objects on the m ap s rep resent containers, and som e rep resent p arts of the stand ard L3 view . Screenshot is from an active installation at MøllerGru p p en in N orw ay, w ith p erm ission to u se by Bjørn Asp rem . N ode View Su p p ort for filtering of nod e view s w as ad d ed in V7.5. This w as based , how ever, on legacy N N M filter d efinitions. V7.51 ad d ed su p p ort for ET top ology filters to be u sed instead . This is m ore efficient, consistent and scalable. If ru nning V7.51 or above, the p roced u re to sw itch to ETbased filters is fou nd in: $OV_DOC/ WhitePapers/ nod eView _ETfilters.pd f In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. 241 Fognet’s Field Guide to OpenView NNM A nod e view that em p loys a filter can be called by u rl: http:/ / "nnm server":7510/ topology/ nod eView ?filter=filternam e V7.51 Interm ed iate Patch 18 fixed a p roblem w ith the URL ip Range sp ecification, so these can now be u sed to qu alify nod e view s, e.g.: http:/ / "nnm server":7510/ topology/ nod eView ?filter=filternam e&ipRange=10.2.*.* Interface View To set Interface View to not show d isabled interfaces, change the inclu d eDisabled p aram eter from 1 to 0 in the follow ing file and restart ovas u sing ovstart/ovstop, bu t only if ru nning versions of N N M p reviou s to 7.53: $OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l If ru nning N N M 7.53, change the inclu d eDisabled p aram eter from 1 to 0 in the follow ing file: $OV_AS/ w ebapp s/ topology/ WEB-IN F/ servletRegistration/ 1IfaceServlet.xm l Then ru n the follow ing scrip t to ad d you r changes to the w eb.xm l file: $OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl Restart ovas u sing ovstart/ovstop. N ode Status and Interface Status View s These view s w ere ad d ed in V7.53 to d isp lay service oriented m anagem ent inform ation abou t nod es and interfaces su ch as w ho is assigned to oversee the nod e, the cu rrent lifecycle state of the nod e, and notes that can be ad d ed by an op erator. These view s are note enabled by d efau lt and requ ire that ET and the APA are enabled . Use the follow ing com m and s to enable or d isable these tw o view s: $OV_BIN/ enableN SVandISV.ovpl $OV_BIN/ d isableN SVandISV.ovpl 242 Dynamic Views Alarm View Introd u ced in N N M V7.53, this tabu lar view is d esigned to rep lace the Java Alarm Brow ser and has all the sam e fu nctionality. The Alarm view requ ires that ET and the APA are enabled . N ote that Container View Access Control (Page 239) ap p lies to Alarm View as w ell. N eighbor View In N N M V7.51 and higher, the d efau lt N eighbor View hop cou nt can be changed . In com p lex netw orks, the d efau lt of “2” m ay create view s that are too com p lex. To change the d efau lt, ed it the follow ing file and change the defaultNumHops valu e and restart ovas u sing ovstart/ovstop, bu t only if ru nning versions of N N M p reviou s to 7.53: $OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l If ru nning N N M 7.53, change the defaultNumHops valu e in the follow ing file: $OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ 1N eighborServlet.xm l Then ru n the follow ing scrip t to ad d you r changes to the w eb.xm l file: $OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl Restart ovas u sing ovstart/ovstop: VRRP View Su p p ort for this view ap p eared in a p atch to V7.5 and a w hite p ap er w as su p p lied w ith V7.51 in $OV_DOC/ WhitePap ers/ VRRP.p d f. This d ocu m entation w as rolled into the Gu id e to Using Extend ed Top ology in V7.53. VRRP View requ ires p u rchase of the Ad vanced Rou ting SPI. VRRP su p p ort by N N M has the follow ing cap abilities: • Discover virtual router groups on d evices that support RFC 2787 • RC-VRRP MIB allow s N ortel d evices to d iscover VRRP groups • FOUN DRY-SN -IP-VRRP-MIB d iscovers virtual router groups Sp ecial VRRP Op enView enterp rise events are generated w ith VRRP p rotocol su p p ort and sp ecial VRRP correlators are also u sed . Check the follow ing LRF flag to netm on to see if it is set to tru e: m igrateH srpVirtualIP=true 243 Fognet’s Field Guide to OpenView NNM If not, set this flag p er the LRF p roced u re on p age 5 and then verify that any VRRP virtu al interfaces have not been ad d ed to the N N M top ology u sing ovtopodump. Rem ove them u sing: ovtopofix –r <router_name>. Interp reting VRRP Grou p Statu s: Unknown (blue) N ormal (green) Warning (cyan) Minor (yellow ) Major (orange) Critical (red ) State und eterm ined The VRRP Group is operational The VRRP group has an interface in the Active State and an interface in the Standby state, but it also has at least one interface that is not in the Listen state. The VRRP Group has an interface in the Active state, but there is no interface in th e VRRP group w hich is in the Stand by state. Multiple interfaces in the VRRP group are in the Active state, or m ultiple interfaces in the VRRP group are in the Stand by state. The group has no interface in the Active state. OSPF View ET‟s OSPF View p rovid es a rep resentation of an OSPF Area. By d efau lt, it show s Area 0 (0.0.0.0). Tables show rou ters filtered by the area, w ith basic d ata inclu d ing neighbors. OSPF View requ ires p u rchase of the Ad vanced Rou ting SPI. OSPF d iscovery u ses RFC 1253 or RFC 1850 MIBS. The RFC 1850 MIB (OSPF V2) is backw ard s-com p atible w ith RFC 1253. A seed IP ad d ress of an OSPF rou ter ru nning the MIB is requ ired to start the d iscovery. Enter this seed rou ter in the configu ration file below . The seed file can be configu red to d iscover via an inclu sive ru le or an exclu sive ru le, bu t not both. Once the seed file is p op u lated , ru n the startu p scrip t and check the log file for errors. OSPF Disco configuration file: OSPF Disco startup script: Error log: Database: $OV_CON F/ nnm et/ Ospf.cfg $OV_BIN/ ospfd is.ovpl $OV_PRIV_LOG/ ospfd is.err $OV_DB/ ospf/ ospfd is.data Path View Path View com p u tes p aths u sing SN MP qu eries in real tim e. Contrast this w ith Problem Diagnosis, w hich u ses p robes and m aintains u sage d ata over tim e (p age 167) and Access Path View , a contribu ted ap p lication also know n as Sm art Path (p age 168.) N N M 7.53 244 Dynamic Views introd u ced the ability to u se ET d ata to p rovid e m ore intelligent p aths. To enable this featu re, set the valu e for restrictToET to true in the follow ing file: $OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ 1IPathServlet.xm l Then ru n the follow ing scrip t to ad d you r changes to the w eb.xm l file: $OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl Restart ovas u sing ovstart/ovstop. The follow ing configu ration file can be u sed to chang e the Extend ed Top ology algorithm s for com p u ting p aths: $OV_CON F/ nnm et/ topology/ NMActiveRoute.Conf The V7.53 Gu id e to Using Extend ed Top ology lists the conf file p aram eters and how they can be changed in Chap ter 6. HSRP View H SRP m anagem ent requ ires p u rchase of the Ad vanced Rou ting SPI. N ote that som e u p grad e p ackages for N N M inclu d ed the Ad vanced Rou ting SPI. H SRP m anagem ent is su p p orted only for Cisco H SRP and H SRP EXT MIBs. Cisco IOS‟s 12.1(14) and 12.2(13) requ ire m anu al configu ration for the tracked interface inform ation (stand by IPs) to ap p ear in the MIB. Old er versions au tom atically p op u late this info. To configu re N N M to m anage H SRP rou ters see the Gu id e to Using Extend ed Top ology. To force an ET Discovery, see the p roced u re on p age 10. To check to see if ET d iscovered any H SRP grou p s: UN IX: $OV_SUPPORT/NM/ovet_topoquery \getAllHSRPs | grep EntityName | cut –d: -f2 Wind ow s: Run: ovet_topoquery > out.txt and search out.txt for EntityNam e, and look for the IP ad d ress on that line Interp reting H SRP Statu s: Unknown (blue) N ormal (green) Warning (cyan) State und eterm ined All stand by routers are available. One IF in the Active state, and one IF in the Stand by state, but also at least 245 Fognet’s Field Guide to OpenView NNM Minor (yellow ) Major (orange) Critical (red ) one that is not in the Listen state. One or m ore stand bys are unavailable. One IF in the Active state, no IFs in the Standby state. No stand by available. Multiple IFs are in Active state, or m ultiple IFs in the Stand by state. Routing functionality likely affected . N o interface in the Active state. Trou bleshooting H SRP: The log file is $OV_PRIV_LOG/ ovet_d aH SRPSnm p .log Managing overlapping address spaces (OAD ) Also know n as Du p IP and OAD, this cap ability is only su p p orted in N N M versions 7.0 or higher and requ ires the AE version . In ad d ition, extend ed top ology d iscovery m u st be enabled . OADs are d iscovered and m onitored by the ovet_poll d aem on and OAD objects are m aintained in the ET d atabases, bu t the APA need not be enabled to m anage OADs. Discovered OAD nod es are not visible to any of the N N M legacy d atabases or GUIs, and thu s are not d isp layed in ovw m ap s. OAD configu ration and trou bleshooting is covered in Chap ter 3 of the Gu id e to Using Extend ed Top ology and in Chap ter of that gu id e in V7.53. The d evices in the ad d ress sp ace m u st be reachable from the N N M server via a d evice that p rovid es Static N AT and those d evices m u st have valid rou te to the N N M server. Statu s can be m anaged on the overlap p ing d evices, only if ICMP an d/ or SN MP are not firew alled along the w ay. In ord er for N N M‟s OAD to p rop erly w ork, the N AT im p lem entation need s to be com p liant w ith the RFC's, not d ynam ic, and not nested . An exam p le of d u p ip .conf can be fou nd in the d irectory: $OV_CON F/ nnm et/ d u p ip . A cu stom ized cop y of this d u p ip .con f need s to be created for each overlap p ing ad d ress sp ace. A u niqu e ad d ress sp ace nam e d efines a su bd irectory u nd er $OV_CON F/ nnm et/ d u p ip so the configu ration file nam e for the ad d ress sp ace nam ed “p hil” w ou ld have to be: $OV_CON F/ nnm et/ d upip/ phil/ d upip.con f The ovdupip com m and can be u sed to test and to qu ery the OAD configu ration. There is no m an/ ref p age for d u p ip .conf in N N M 7.01 or 7.5, bu t the file itself has very good help . There is a m an/ ref for ovdupip. 246 Dynamic Views Seed files, as exp lained in the d u p ip .conf file, are necessary for p rop er d iscovery of the OADs by ET. Once the configu ration has been valid ated u sing ovdupip, “refresh configu ration” m u st be selected in the OAD section of the ET configu ration GUI, then d iscovery for the new OAD can be initiated in the d iscovery section. Use the below com m and to sp ecify an overlap p ing ad d ress d om ain ID (OADID) for an entity to be p olled . If this op tion is u sed then the nod enam e or virtu alIP is assu m ed to be a p rivate IP ad d ress. The OADID m u st be the ID and not the overlap p ing d om ain nam e. To m ap the nam e to an OADID, p erform the com m and ovdupip -i a: ovet_demandpoll.ovpl –r <OADID> Any nod es listed in the OAD Configu ration file shou ld also be entered into the netmon.nod iscover file and existing N N M-d iscovered nod es m u st be p u rged from the IPMAP top ology for the OAD to be p rop erly d iscovered by ET. When the OAD is d iscovered , the OAD table d isp lay show s the nod e nam e, the m anagem ent IP ad d ress, the p rivate IP ad d ress, the rou te d istingu isher, and th e nod e‟s statu s. In the event brow ser, the ad d ress sp ace nam e is p ostfixed to the m anagem ent IP ad d ress. On Wind ow s, w hen m igrating from N N M version 7.0 to 7.5x, ru n the follow ing com m and to u p d ate OAD zone ID‟s to the new 7.5x form at: %OV_BIN%\migrate70to701ZoneNumber.ovpl In N N M 7.0, d eleting an OAD (Overlap p ing Ad d ress Dom ain) requ ired ru nning a fu ll d iscovery. In N N M 7.01, a fu ll d iscovery is no longer necessary. D elete an overlapping address spaces (OAD ) Follow this p roced u re to d elete an OAD called “haze” w ithou t initiating fu ll d iscovery: 1. 2. 3. 4. 5. 6. 7. Go to the $OV_CON F/ nnm et/ d upip/ haze d irectory Ed it “d upip.seed ” file and com ment out all entries using a “#” Go to the ET Config GUI Select the “Overlapping Add ress Domains” tab Press the “Refresh Configuration and Activate Changes” After a m essage ind icating success, press the “Continue” Select “haze” OAD from the table and press “Discover Zone” 247 Fognet’s Field Guide to OpenView NNM 8. 9. 10. 11. 12. 13. 14. 15. Wait until “Please wait” m essage d isappears, then press “Continue” Wait to see a “Discovery Com plete” pop -up m essage from Hom e Base Go to OAD View from Hom e Base, and confirm “ haze” is not in the Table Delete d irectory "$OV_CON F/ nnm et/ d u pip/ haze" and its contents Go to the ET Config GUI Select the “Overlapping Add ress Domains” tab Press the “Refresh Configuration and Activate Changes” After a m essage ind icating success, p ress the “Continue” N N M 7.0, d eleting an OAD (Overlap p ing Ad d ress Dom ain) requ ired ru nning a fu ll d iscovery. In N N M 7.01, a fu ll d iscovery is no longer IPV6 N N M‟s IPV6 su p p ort is an ET featu re - IPV6-d iscovered d evices are not visible u nd er legacy N N M d atabases or GUIs. IPV6 d evices are d iscovered and p olled via ICMPv6 p ing, and IPV6 d iscovery is not enabled by d efau lt. IPV6 d iscovery can be enabled by ru nning: setupExtTopo.ovpl The IPV6 d ynam ic view s in N N M 7.5 and higher show s layer 3 connectivity. Licenses are requ ired for both IPV4 (N N M) & IPV6 (ET), and only d u al-stacked (IPV4 & IPV6) rou ters are su p p orted (i.e. t he su p p orting rou ter m u st be d iscovered via netmon first). H itachi, N EC, Ju nip er and Cisco equ ip m ent is su p p orted , and IPV6 d iscovery is only su p p orted on H P-UX 11.11 and Solaris 8 & 9 N N M installations. Only RFC 2465-su p p orted d evices can be fu lly m anaged (i.e. p rop erly m ap p ed in top ology), bu t v6 end -nod es are d iscovered and p olled for statu s via ICMPv6 p ing. An ad d itional license for the IPV6 nod es is requ ired . For H P-UX, version 11i is requ ired along w ith the ap p rop riate OS bu nd les. More info can be fou nd u nd er the IPV6 SPI for N N M at H P's w eb site. Configu re IPv6 d iscovery u sing the p roced u re ou tlines in the Gu id e to Using Extend ed Top ology. There, the follow ing five control files in the $OV_CON F/ nnm et d irectory are d ocu m ented : IPv6.conf IPv6Polling.conf IPv6Prefix.conf IPv6Scope.conf IPv6Seed .conf The follow ing IPV6 Statu s events are su p p orted in N N M 7.x: - Ad d ress DOWN 248 Dynamic Views - Ad d ress UP (log only) - Interface Status Change (log only) - N od e Status Change (log only) - Prefix Group Status Change (log only) There is a Pair-w ise correlation for Ad d ress UP and Ad d ress DOWN . If Ad d ress UP occu rs less than 10 m inu tes after a corresp ond ing Ad d ress DOWN , both events are su p p ressed . See p age 124 for m ore abou t this correlation. If an IPv6 nod e is d ow n or u nreachable d u ring N N M Extend ed Top ology d iscovery, N N M m ay not inclu d e that nod e in the ET d atabase (d ep end ing on the p resence of that nod e's ad d resses in the ARP caches of IPv6-resp onsive nod es). This d iffers from the behavior of IPv4 nod es, w hich w ill be treated as "u nresp onsive" nod es d u e to their p resence in N N M's d atabases. On Solaris p latform s, if IPv6 nod es are not configu red w ith hostnam es, pmd p erform ance w ill su ffer as the m onitorIPv6Agent send s (m any) statu s events abou t these nod es. To ad d ress this issu e, configu re the ip N oLooku p cache to p revent these looku p s by: 1. 2. 3. Discover IPv6 d evices Ad d the ad d resses d iscovered to a file called <filenam e> Run snmpnolookupconf -file <filename> Port Admin tool This tool introd u ced in V7.53 p rovid es a GUI for d isabling/ enabling APA statu s p olling to p orts of interfaces that form erly requ ired u sing the ovet_toposet.ovpl com m and . To lau nch the tool, select a target nod e in any Dynam ic View and choose m enu selection Ed it -> ET Monitoring Manager -> Port Ad m in Tool. Aggregated port support Su p p ort for aggregated p orts in ET started w ith V7.5, w here m u ltip le p hysical p orts on d evices that su p p ort Cisco PAgP w ou ld be rep resented via tru nk virtu al p orts. V7.51 ad d ed su p p ort for N ortel‟s MLT and SMLT via the ovet_d aBaySSw (BaystackSw itchAgent) and ovet_d aPassSw (Passp ortSw itch) d evice agents. For m ore inform ation on su p p ort for N ortel aggregated p orts, see the follow ing w hite p ap er: $OV_DOC/ Whitepapers/ MLT.pdf 249 Fognet’s Field Guide to OpenView NNM In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using Extend ed Top ology u ser m anu al. For d etails on how aggregated p orts are hand led by the APA, see Page 77. Su p ort for d iscovery of aggregated p orts on Alcatel Tim etra and Riverston sw itches w as ad d ed via new d evice agents in Ju ne, 2007. See Page 219 for how to check for latest d evice agents. Increase JAVA heap size In large environm ents, increasing JAVA heap size w ill help red u ce ru nning ou t of m em ory issu es that m ay crop u p . Ad d the follow ing LRF flag “-Xm x384m ” to ovas.lrf p er the p roced u re on Page 5. This sp ecifies a heap size of 384 Meg. The m ax is 512m bu t this is not recom m end ed and the d efau lt is 128 MB. Add, delete, manage or unmanage objects via URLs This hand y featu re w as ad d ed in N N M 6.41 to p rovid e a rem ote facility to rep lace loadhosts, ovtopofix and other assorted local facilities for m anip u lating objects. The URLs are: http:/ http:/ http:/ http:/ / / / / <N N M-Server>::7510/ <N N M-Server>::7510/ <N N M-Server>::7510/ <N N M-Server>::7510/ topology/ ad d topology/ d elete topology/ manage topology/ unmanage These URLs are configu red via the follow ing xm l files: All: $OV_AS/ w ebapps/ topology/ WEB-IN F/ d ynam icView sUsers.xm l Prior to NN M 7.53: $OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l 7.53: …w ebapps/ topology/ WEB-IN F/ servletRegistration/ 1ManageServlets.xm l 250 Troubleshooting dynamic view s The first p lace to look w hen trou bleshooting Dynam ic View s issu es is the JAVA console. On UN IX use: On Wind ow s: / opt/ java1.4/ jre/ binControlPanel Control panel -> Java Plug-in Alw ays confirm the Web server is ru nning. On UN IX: On Wind ow s: ps –ef | grep ovhttpd Control panel ->Ad m in Tools -> Internet Inform ation Manager -> Web server Confirm there is a m atch betw een top ology view s and top ology d atabases. For m ore inform ation on ovet_top oqu ery, see p age 91. For m ore inform ation on ovtop od u m p see p age 67. Several p roblem s w ith d ynam ic view s m ay ar ise d u e to JAVA caching. Caching can be safely d isable from the JAVA control p anel m entioned above. D ynamic view s update issues due to caching There are several layers of caching p ossible that cou ld lead to stale top ology inform ation (d u e to lack of u p d ate) in d ynam ic view s. Brow ser cache: For IE, clear via Tools -> Internet Options ->General tab -> Tem porary Internet files -> Delete Files. Proxy servers: These also often d o caching. The best practice is to d isable. For IE d o the follow ing: Tools -> Internet Options -> Connections tab -> LAN Settings -> Uncheck "Use a proxy server for your LAN ”. Java Plu g-in Cache: On Wind ow s, navigate: Start -> Control Panel -> Java Plug-in -> Cache tab -> Clear. On UN IX, brow se to ${JPIDIR}/ jre/ ControlPanel.htm l, w here ${JPIDIR} is the location w here the Java Plug-in is installed . The cache can be cleared at this point. Web content op tim izers or internet accelerators: 251 Fognet’s Field Guide to OpenView NNM These network d evices can hijack URL requests and serve content that is stale. In most cases, these servers can be configured to pass either requests from certain clients or content from certain servers. D ynamic view s registration files Extensions to d ynam ic view s are ad d ed u sing XML files u nd er: $OV_WWW_REG/ d ynam icView s/ C/ H P m enu s are in d ynam icView s.xm l (d o not m od ify this file). All files get m erged au tom atically into m enu settings.xm l located in: $OV_WWW/ htdocs/ $LAN G/ d ynam icView s Many trad ition al ovw m enu settings are only visible to d ynam ic view s that are lau nched locally. Most SN MP grap hing tools u nd er the p erform ance m enu bar fall into this category. To enable these m enu item s to be visible to those w ho lau nch d ynam ic view s from rem ote brow sers, change the localOnly flag from tru e to false for each m enu item in the d ynam icView s.xm l file. D ynamic view s and edge connectivity Dynam ic View s has sp ecial su p p ort for d isp laying layer 3 ed ge rou ter connectivity across WAN s in V7.5 and later. In that version, p oint-top oint connectivity is ad d ressed , bu t not p oint -to-m u lti-p oint. Exam p les of su p p orted p rotocols inclu d e ATM, Fram e Relay, t1/ t3 and Sonet. Typ ically, su ch connections w ill be tw o -nod e su bnets w ith 30-bit su bnet m asks, and this is configu rable in the follow ing file: $OV_CON F/ nnm et/ Ed ge3Conn.cfg Transfer MIB application builder apps to D ynamic view s Follow the step s below to transfer ovw Menu bar ap p lications that w ere bu ilt u sing the MIB Ap p lication Bu ild er into Dynam ic View s: 1. To invoke MIB App lication Build er app through a “trad itional” web interface; use: http:/ / nnm server:3443/ OvCgi/ snm pview er.exe Select MIB Application Build er application, select “Display in N ew Wind ow.” In new w indow, right click and look at Properties, e.g.: http://nnmserver.xx.com:3443/OvCgi/webappmon.exe?ins=res new&sel=somenode.xx.com&app=IP+Tables&act=Disk+Space&arg 252 Dynamic Views =helpBrowser+ovw%3aHPdiskSpace+%09%09++++++appendSelectL ist+appendSelectListToTitle+%09%09++++++headingLine+1+co mmandTitle+%22Disk+Space%22+%09%09++++++iconName+%22Disk +Space%22+%09%09++++++cmd+rbdf&cache=2795 2. In m enu item s XML file, copy the URL from properties w indow and m od ify slightly, as below : <!-- A C T I O N S --> <Actions> <action actionId=―Disk Space‖ url=―http://${localhost}${localport}/OvCgi/webappmon.exe?i ns=resnew&sel=${names}&app=IP+Tables&act=Disk+ Space&arg=helpBrowser+ovw%3aHPdiskSpace+%09%09++++++ap pendSelectList+appendSelectListToTitle+%09%09++++++heading Line+1+commandTitle+%22Disk+Space%22+%09%09++++++iconName+ %22Disk+Space%22+%09%09++++++cmd+rbdf&cache=2795‖/> </Actions> 3. Define <m enu> entry to use new action . Customize dynamic view s-based alarm brow ser The follow ing file can be u sed to configu re the JAVA -based alarm brow ser: $OV_WWW/ conf/ N N M.spec Exam p les of configu ration entry p oints inclu d e color d efinitions, filter grou p and action d efinitions. A u sefu l cu stom ization to this file is to d isable access to the event correlation configu ration screens . Do this bu t com m enting ou t the follow ing line: CATGETS(14, 11, "Event Correlation Configuration") f.action "a_id 11"; After m aking changes to the N N M.sp ec file, stop and retart the ovalarmsrv d aem on and refresh any op en brow ser view s. 253 22. Databases and Reporting There are several d istinct d atabases u sed w ithin recent versions of N N M. Som e are relational and som e are flat file d atabases. The N DBM flat file op erational d atabases inclu d e: m ap , object, top ology, snm p Collect, Binary Event Store. The SN MP configu ration d atabase: ovsnm p .conf, is not consid ered an op erational d atabase. It is com p osed of 5 N DBM d atabase files, three of w hich hold p olling and SN MP configu ration d ata, and the other tw o are caching d atabases. cachedb hold s a list of hostnam e to IP ad d ress m ap p ings and IP ad d ress to hostnam e m ap p ings; nolookupdb hold s hostnam es that w ill not be resolved to an IP Ad d ress. Problem Diagnosis (PD), w hich has been bu nd led w ith N N M AE since version 6.4, stores p robe d ata in a text file d atabase. Relational d atabases u sed by N N M inclu d e the d ata w arehou se and the Extend ed Top ology DB. The d ata w arehou se (DW) hold s exp orted trend , top ology, and event d ata for relational u se by the rep orting su bsystem . This is also referred to as the “em bed d ed ” d atabase, and it u ses the Solid DBMS OEM‟d from Solid tech. Extend ed Top ology u ses the sam e Solid DB instance as the DW, bu t is a sep arate d atabase that is not exp osed (the schem a is not available). The DW p art of the Solid DB can be changed to u se Oracle (or on Wind ow s, MSSQL) instead , bu t the ET p art cannot. D ata w arehouse Solid DB gets d ata via au tom atic exp ort of Events, snmpCollect and top ology d ata. The exp ort hap p ens throu gh the ovrequestd d aem on‟s internal sched u ler. The com m and request_list schedule show s the exact tim ing for each exp ort in UN IX cron form at). Events get exp orted every 30 m inu tes, top ology tw ice d aily, and trend d ata every 40 m inu tes. The exp orts can be changed by ed iting the request.properties files in the su bd irectories u nd er: 254 Databases and Reporting $OV_AN ALYSIS\ ovrequestd \ requests\ C\ export Events are exp orted to the nnm _event_d etail table, snmpCollect d ata is exp orted to the snm p _raw _trend table. The follow ing d irectory contains the Solid DB backu p s, w hich can be entirely rem oved if there is a need to recover d isk sp ace: $OV_DB/ analysis/ d efault/ backup You can stop Solid from d oing backu p s by ed iting solid .ini in $OV_DB/ analysis/ d efau lt and com m enting the follow ing line w ith a sem icolon: At=23:00 backup After m aking changes to solid .ini, ru n ovstop ovdbcheck and then ovstart ovdbcheck. Solid database Details for the d ata w arehou se‟s u se of Solid are given above. The extend ed top ology d atabase is on the sam e instance of Solid bu t it cannot be qu eried w ith ovdwquery. The ET d atabase schem as are not p u blicly available. ET d ata can be qu eried , how ever, u sing: $OV_SUPPORT/ N M/ ovet_topoquery Solid DB d ocu m entation: http:/ / w w w.solid tech.com / d evresources/ d ocum entation.htm l Solid ODBC Drivers: $OV_NEW_CONF/OVDB-RUN/conf/analysis (UNIX), or $OV_WWW/htdocs/classes/solid (Wind ow s), or $OV_CONF/analysis/reportTemplates/Solid/driver, or http:/ / w w w.solid tech.com / library/ software.htm l D ata w arehouse queries Qu eries can be m ad e to the d ata w arehou se d ata via the ovdwquery com m and . SQL com m and s can be d irectly entered , or p laced in a file, and then called in w ith the –file op tion. The d efau lt u ser and p assw ord are show n in the below exam p le qu ery: ovdwquery -u ovdb -password ovdb -file router.query 255 Fognet’s Field Guide to OpenView NNM The –x pwdfile <password file> w as ad d ed and the –securefile op tion w as u p d ated in N N M V7.51 to m ake this com m and N IAP com p liant. Thu s –u and –p assw ord op tions show n above w ere d ep recated . See the m an/ ref p ages for u sage d etails. The Solid DB schem a can be u nraveled by looking at the table creation scrip ts in: $OV_CONF/analysis/sqlScripts/tables_* The follow ing p red efined bu ilt-in p aram eters m ay be u sed in the SQL WH ERE clau ses: $NOW, $BEGIN_TODAY, $BEGIN _YESTERDAY, $BEGIN_WEEK, $BEGIN_LAST_WEEK, $BEGIN_MON TH , $BEGIN_LAST_MON TH , $BEGIN_YEAR, and $BEGIN_LAST_YEAR Som e exam p les of SQL qu eries follow : Retu rn all nod es by hostnam e: SELECT ip_hostname FROM nnm_nodes; Retu rn all nod es by SN MP sysnam e: SELECT sysname FROM nnm_nodes; Retu rn all m anaged interfaces from certain hosts: SELECT a.snmp_ifphysaddr, b.ip_hostname, c.ip_address FROM nnm_objects a, nnm_nodes b, nnm_interfaces c, nnm_objects d WHERE c.topo_id = a.topo_id and c.node_id = d.ovw_id and d.topo_id = b.topo_id and b.ip_hostname like 'S%lab' and a.ip_status != 5; Retu rn all m anaged interfaces m atching SN MP ifDescr: SELECT a.snmp_ifphysaddr, c.ip_address FROM nnm_objects a,nnm_interfaces c WHERE c.snmp_ifdescr = 'Compaq Ethernet/FastEthernet or Gigabit NIC' and c.topo_id = a.topo_id; Retu rn tod ay‟s exp orted events: SELECT * from nnm_event_detail WHERE event_timestamp > $BEGIN_TODAY; Retu rn only IP ad d ressed nod es and interfaces: SELECT n."ip_hostname", n."sysname", i."snmp_ifname", i."ip_address" FROM "ovdb"."nnm_nodes" n,"ovdb"."nnm_objects" o, "ovdb"."nnm_interfaces" i WHERE n."topo_id" = o."topo_id" and o."ovw_id" = i."node_id" 256 Databases and Reporting and i."ip_address" <> '0.0.0.0'; Solid database performance In highly-scaled environm ents, there is no p erform ance ad vantage to sw itching from Solid to Oracle. Oracle hand les sim u ltaneou s access better than Solid , bu t overall, Solid is faster than Oracle p er H P ‟s d ocu m entation. In highly-scaled environm ents, be su re that the Solid DB is not on the sam e d rive as the OS. Using Raid 0 or som e other RAID in com bination w ith m oving Solid off the OS d rive can facilitate an increase in p erform ance of 10% or m ore. With the release of V7.51, a w hite p ap er ap p eared on how to im p rove Solid d atabase p erform ance in the $OV_DOC/ WhitePap ers d irectory. It is not clear from the w hite p ap er w hether the recom m end ations p rovid ed can ap p ly to earlier version s of N N M/ Solid . Som e of them are know n to d o so. All of the below p aram ers are set in the follow ing file: $OV_DB/ analysis/ d efault/ solid .ini For CacheSize, the w hite p ap er recom m end increasing this from the d efau lt of 10MB to 512MB. The m inim u m is 512KB and there is no m axim u m , bu t H P recom m end s the m axim u m not exceed the am ou nt of p hysical m em ory. The Thread s d efau lt is 5, w hich H P says is ad equ ate for system s w ith one or tw o p rocessors. Increase Thread s if the N N M server has m ore than tw o p rocessors. A com m on recom m end ation is to set thread s to (2x (nu m CPU's) + 1). Setting ForceThread sToSystem Scop e to “yes” on N N M Servers ru nning Solaris can d ram atically im p rove p erform ance on that p latform only. After m aking changes to solid .ini, stop and restart all op enview p rocesses. D etermining the version of Solid On UN IX: what /opt/OV/bin/ovdbrun |grep Embedded On Wind ow s: find /i "embedded" "%OV_DB%\analysis\default\solmsg.*" D atabase command utilities Data w arehou se configu ration tools: 257 Fognet’s Field Guide to OpenView NNM ovdbcheck ovdbdebug ovdwconfig.ovpl ovdbsetup ovdwloader ovdwunloader Start, stop or monitor DW Valid ates DB connections Configure ODBC source, security Integrate DB into existing Oracle Reload data into DW Copies DW to special file or to CSV Raw d atabase m aintenance tools: ovcoltosql Convert snmpCollect data to trend ovcoldelsql Red uce or d elete raw trend d ata ovcolqsql Retrieve and sum marize trend data ovdwtopo Retrieve topology data to DW ovdwtrend Export, sum and trim s trend data (com bines ovcoltosql & ovcoldelsql) ovdwevent Export, trim event to/ from DW ovdweventflt Restrict events exported to DW Reporter URLs Use the follow ing URLs to access the N N M rep orting interface: http:/ / nnm server[:port]/ OvCgi/ nnm RptPresenter.exe http:/ / nnm server[:port]/ OvCgi/ nnm RptConfig.exe nnmRptConfig.exe is the p rogram u sed to configu re rep orts and nnmRptPresenter.exe is the p rogram to view w eb rep orts. Connecting Crystal Reports to N N M (Window s) Connecting N N M rep ort d ata to Crystal Rep orts r equ ires Crystal Rep ort 9 Professional version or greater. To d o this, follow these step s: 1. Copy stcw3230fe.d ll to C:\ w innt\ system32 on NN M system . 2. Go to ODBC in System DSN Datasource nam e : ovd brun Description : N etwork Nam e : tcpip <NN M server IP> 2690 3. Within Crystal Reports: Select create a new connection -> ODBC (RDO) usernam e : ovd b passw ord : ovd b Using Oracle for the data w arehouse The m ost com m only asked qu estion p osed by those w ho are trying to d ecid e w hether to u se Oracle for N N M‟s d ata w arehou se in p lace of Solid is: “Can N N M share the sam e DB instance w ith an OVO 258 Databases and Reporting installation?” The N N M DW can ind eed share an OVO Oracle instance, w hich is nam ed “op enview ” by d efau lt. A com m on p roblem w ith Oracle as DW for N N M in large-scale environm ents is that the ovdb.nnm_event_varbinds table tend s to fill u p . To extend this table, issu e the follow ing SQL com m and : ALTER TABLE ovdb.nnm_event_varbinds storage maxextents unlimited; See the “Managing” gu id e for instru ctions on changing from Solid to Oracle for a given p latform . In su m m ary, the follow ing com m and s are u sed to m anage the sw itch: ovdbsetup ovdbcheck ovdbdebug On UN IX p latform s, if the Oracle SID is set to anything other than the d efau lt “op enview ,” su bsequ ent invocations of ovdbsetup m ay generate errors. To correct this, m ake su re the correct valu e of $ORACLE_SID is p laced in the listener.ora file, located as listed below : H PUX: SOLARIS: LIN UX: / etc/ listener.ora / var/ op t/ oracle/ listener.ora $ORACLE_H OME/ netw ork/ ad m in/ listener.ora N N M d oes su p p ort 64-bit Oracle d atabases. A com m ent in the Su p p ort Matrix references that Oracle is only su p p orted in 32-bit versions on H P-UX and Solaris. This statem ent w as rem ov ed in the N N M 7.5 Release N otes. It is not the case that the Oracle d atabase server is 64bit. Som e versions of 64-bit Oracle d id not d eliver 32-bit Oracle d atabase d rivers. Given that N N M is 32-bit, N N M w ou ld not have a valid d atabase d river to op erate, bu t N N M now relies u p on a d atabase d river p rovid ed by the ODBC vend or called OVoraWire. With this d river, N N M is assu red that a 32-bit d river exists. OD BC D river Manager Version errors When u sing a external DB u nd er Wind ow s, the follow ing ODBC Driver Manager p op -u p w ind ow for a version m ism atch for variou s ODBC u tilities m ay occu r, for exam p le: The ODBC resource DLL (C:\WINNT40\System32\odbcint.dll) is a different version than the ODBC driver manager (C:\WINNNT40\System32\ODBC32.dll). 259 Fognet’s Field Guide to OpenView NNM If these occu r, reinstall the ODBC com p onents to ensu re p rop er op eration by ap p lying the latest Service Pack. This m ism atched DLL p roblem is a know n issu e w ith Wind ow s and is not u niqu e to N N M. See Microsoft Know led ge Base article: Q170769 for m ore inform ation. Accessing D W data directly from Microsoft Excel Install the solid ODBC d rivers (see above) on a Wind ow s client ru nning Excel. In the control p anel create a d ata sou rce w ith the d river and the m achines d etails. From Excel, select: Data -> Get External Data -> N ew Database Qu ery. Select the d ata sou rce created above and follow the p rom p ts. D ata w arehouse compatibility: N N M 6.0 to N N M 6.1 The ODBC interface betw een N N M and Solid changed from N N M 6.01 to N N M 6.1. In 6.01, shared m em ory w as u sed , in 6.1 TCP/ IP is u sed . N N M 6.01 (Wind ow s) w as the last version to p rovid e Excel tem p lates and N N M 6.1 introd u ced the w eb-based rep orting infrastru ctu re. Excel tem p lates no longer w ork after N N M 6.1. D isable the data w arehouse Disabling the d ata w arehou se is not a catastrop hic o p eration. In fact, it can greatly im p rove the p erform ance of the Solid DB in highly -scaled environm ents. The only consequ ence to d isabling the DW is that rep orts w ill have m issing d ata. To d isable the d ata w arehou se, ru n: ovstop ovrequestd ovdelobj $OV_LRF/ovrequestd.lrf More inform ation abou t LRF files can be fou nd on p age 5. The scrip ts that p op u late and aggregate this d ata are: ovdwquery, ovdwevent and ovcoltosql These scrip ts are called by ovrequestd d aem on to p op u late the d ata w arehou se. The p op u lation of d ata into the d ata w arehou se can also be stop p ed by com m enting ou t the at job in the follow ing file: %OV_DV/analysis/default/solid.ini Rebuild the data w arehouse Ru n the below com m and on any version of N N M above v6.1 to rebu ild the d ata w arehou se from scratch. All d ata w ill be cleared in this p rocess, and the d efau lt tables w ill be p op u lated . This is an easy w ay 260 Databases and Reporting to recover d isk sp ace. N ote that ru nning this com m a nd w ill stop all Op enView p rocesses, so m ake su re u ser sessions are closed . $OV_BIN/ovdwconfig.ovpl -type embedded –reload Rebuild the ET D atabase Ru n the below com m and on any version of N N M above v6.1 to rebu ild the Extend ed Top ology d atabase, w hich is a sep arate SOLID d atabase. Stop p ing the Op enView d aem ons w ill cau se this scrip t to fail, so leave them ru nning: $OV_SUPPORT/ N M/ ovet_reloadTopoDBTbls.ovpl Re-ru n setupExtTopo.ovpl after the above com m and su ccessfu lly com p letes. D isabling N N M 6.2 default data collections After version 6.2 of N N M, d efau lt rep orting configu rations w ere a lot less consu m p tive of system and d isk resou rces. Bu t v6.2 in p articu lar had lots of d efau lt d ata collections and rep orting enabled . To d isable these collections and rep orts, invoke the Rep ort Config GUI either throu gh the ovw GUI or via the u rl: http:/ / <nnm_server>:8880/ OvCgi/ nnm RptConfig.exe or http:/ / <nnm_server>/ OvCgi/ nnm RptConfig.exe Dou ble-click 'View Selected Rep orts', select a rep ort and p ress 'stop'; repeat for all of the reports listed. Exit this GUI and invoke the data collections and thresholds GUI, either from the ovw GUI or from the command line via xnmcollect. Select the collections that are “collecting” by default and “suspend” them. See the procedure below for recovering snmpCollect database disk space if desired. Follow the p roced u re below to com p letely rem ove all the SN MP collected d ata. N ote the com m and snmpColDump can be u sed to selectively p are the snmpCollect d atabase as w ell, and exam p les of those com m and are fou nd in the m an/ ref p ages: Run: ovstop snmpCollect Delete all files und er: $OV_DB/ snm pCollect/ $OV_DB/ snm pCollect/ stringData/ Run: ovstart snmpCollect 261 Fognet’s Field Guide to OpenView NNM Extend and customize router performance reports Cisco Rou ter General Perform ance Rep orts (CPU and Free m em ory), are lim ited in the scop e of d evices w hich this rep ort ap p lies to. N ote that in the rep ort configu ration, the "SysObjectID" field is .1.3.6.1.4.1.9.1.* w hich su p p orts only som e Cisco d evices. To inclu d e other d evices, m od ify the follow ing line in the availability.ovp l scrip t: $report_qualifier = '.1.3.6.1.4.1.9.1%'\n" "AND node.snmp_sysobjid LIKE Extending the number of instances in the TopN report To accom p lish this, change the topn_count valu e in: $OV_CONF/analysis/global.conf Support and contributed reporting tools Som e u sefu l rep orts can be d erived from d ata p rod u ced by the scrip ts in the su p p ort and contrib d irectories. The su p p ort d irectory scrip ts in p articu lar contain som e interesting top ology d ata d u m p tools. Su p p ort Directory: getConnected Port.sh (UNIX) m ineETDB.ovpl ovet_topoconnd um p.ovpl Dum p connected port info Gather d evice d iscovery d ata Dum p ad d ress and entity totals Contrib Directory: TopoInventory.ovpl Trend d irectory w riteSelList Event Directory Various topology reports Various SQL for trend reports Dum ps OVw Selections to file Various event reports D atabase maintenance Use the follow ing com m and s to m aintain the d atabases that feed the d ata w arehou se: ovdwevent, ovdwtrend, and ovdwtopo N ote that ovdwtrend fu nctions. com bines ovcoltosql ovdwtrend -trim ovdwevent -trim ovdwevent –trimdetail 0 262 and ovcoldelsql rem ove all from SN MP tables rem ove old est from events rem oves all event table data Databases and Reporting D elete old reports This can be d one from the rep ort configu ration GUI, or by m anu ally d eleting files located in: $OV_SH ARE_HTDOCS/ C/ nnm / reportPresenter/ The follow ing com m and s are u sefu l for selectively d eleting old er rep orts on Wind ow s: cd %OV_WWW%, then rep lace yyyym m w ith year and m onth below and ru n the 2 follow ing com m and s : for / f "d elim s=#" %a in ('d ir d aily.yyyym m *.* / s / b') d o d el "%a \ *.*" / q for / f "d elim s=#" %a in ('d ir d aily.yyyym m *.* / s / b') d o rm d ir "%a" / q If u sing som e regional setting other than EN -US, you m ight end u p w ith several cop ies of each rep ort, and this eats d isksp ace, not t o m ention tim e need ed to ru n ovbackup. Rebuild D W after trim to recover space (UN IX) This p roced u re is available in Versions 6.1 and higher: $OV_CONTRIB/NNM/dataWarehouse/repackDb.sh This scrip t resizes the Solid d ata w arehou se by trim m ing u nu sed sp ace taken by the d atabase. Since the Solid d atabase never gives back any d isk allocated w hen reaching new high -w ater m arks, this scrip t allow s recovery of u nu sed d isk sp ace after trim m ing old d ata. This scrip t: 1) backs u p the d atabase (ju st in case), 2) u nload s the d atabase into a flat file, 3) rem oves and recreate an em p ty d atabase, 4) reload s the d atabase and 5) cleans-u p . To lim it size of Solid DB in N N M V6.1 or V6.2, see H P‟s SSO d ocu m ent nu m ber KBRC00004254 at: openview.hp.com/ sso/ ecare/ getsupportd oc?docid=KBRC00004254 Rem ove Defau lt Rep orts: ovstop ovrequestdovdumpevents Delete $OV_CON F/ analysis/ requests/ C/ * Delete $OV_AN ALYSIS/ ovrequestd / requests/ * Delete $OV_WWW/ htd ocs/ C/ nnm / reportPresenter/ Reports/ * ovstart -v ovrequestd Disable Defau lt Rep orts: Backup snmpRep.conf, ed it and com m ent all lines Delete/ recreate Solid d ata w arehou se and ET DB: $OV_SUPPORT/redoSolid.ovpl 263 Fognet’s Field Guide to OpenView NNM Unload DW d ata to flat files: ovdwunloader -file trend_output_file -trend -v ovdwunloader -file event_output_file -event -v Disable Solid : Prior to N N M 6.41, Solid w as only u sed for DW op erations and cou ld be com p letely d isabled w ithou t m u ch im p act to N N M op erations. More recent versions u se Solid for the Extend ed Top ology d atabase and d isabling Solid w ou ld rend er all ET fu nctions u seless. Proced u res for com p letely d isabling Solid vary w ith N N M version and p latform . Maintain/ p reen w eb rep orts: The w eb rep orts are stored in : $OV_SH ARE_HTDOCS / C/ nnm/ reportPresenter/ Reports Su bd irectories hold rep orts by typ e, then by tim e p eriod . If the su bd irectory is d eleted for any p articu lar d ay, for exam p le, that d ay sim p ly is no longer be available in the Web rep ort. Once d eleted , the rep orts cannot be recreated . Configu re DW, connect to ODBC: ovdwconfig.ovpl Resolving errors w ith database maintenance programs If the follow ing error is seeen: Error: "Data warehouse maintenance program (ovcoldelsql) exited with a non-zero return code of 1 and the following message: Data specified to trim has not been aggregated" This is a non-critical error that ind icates an u nsu ccessfu l d ata-d eletion from the em bed d ed d atabase or an u nsu ccessfu l attem p t to d elete d ata that‟s not p resent in d atabase. In m ost cases, it can be ignored . Extreme filtering in event data exports Only the follow ing events are exp orted by d efau lt: nod e u p , nod e d ow n, interface u p , interface d ow n, nod e ad d ed , nod e d eleted , threshold violation and threshold rearm . To exp ort all alarm s, create an em p ty file in the $OV_CON F\ analysis su b-d irectory called : N O_EXTREME_EVEN T_FILTERIN G Make su re that no extension is ap p end ed to this file nam e , then ru n the follow ing com m and to enable it: 264 Databases and Reporting ovdweventflt -n '*' -o '*' -i N D atabase size limitations Raw d atabases: - The binary event store DB (BES) d efault size is 16 MB and it can grow up to 32 MB. - The SnmpCollect d atabase can grow w ithout bound s, and can be trim m ed w ith the ovdwtrend or snmpColDump com m and s - Topology, Map, and Object d atabases are bound less, but in practice rarely grow beyond a few d ozens of MBs. Solid DB Solid DB can be extend ed u p to abou t 16 GB (2x8 files of 2GB). 16GB is the d efau lt m axim u m d atabase size in V7.5. Increase size of the binary event store (BES) The BES is also called the event d atabase or eventdb. More inform ation can be fou nd in the ov_event m an/ ref p ages. Do not attem p t to d ecrease the size of eventdb, as this m ay cau se the d ata w arehou se to fail. To increase the BES beyond the d efau lt m axim u m of 32 MB, set the follow ing LRF sw itch (p age 5) on pmd to increase the size (in this exam p le to 128MB): -SOV_EVENT;b128 If the op tion to log events to trap d .log is also selected (see p age 100), u se the “-l” (ell) op tion to increase the m axim u m size of that file, for exam p le: -SOV_EVENT;t;l128;b128 If red u cing the size of the BES, clear the eventd b (see below ) before restarting Op enView p rocesses. Clear the event database (BES) To clear the contents of the event d atabase, issu e and ovstop com m and , then d elete the OV_DB/ eventd b d irectory. Restart N N M u sing ovstart. The eventd b d irectory w ill be re-created au tom atically. D ump list of managed nodes into CSV file (UN IX only) This is a typ ical exam p le u se of ovtopodump: ovtopodump | awk '/NODES/, length < 2' | grep -v Unmanaged | grep IP | awk '{print $3}' | sort –u 265 Fognet’s Field Guide to OpenView NNM D ump a clean list of all managed nodes (UN IX only) In the com m and below , d o not su bstitu te a hostnam e w here it says H OSTN AME, u se that exact text: ovtopodump -lr | grep '^HOSTNAME' | sed -e 's/^HOSTNAME:[ ]*//g' | sort D ump interface list for all nodes (UN IX only) for a in `ovobjprint –a ―Selection Name‖ isNode=1 | awk ‗{print $2}‘|sed –e ‗s/‖//g‘|sort` do print $a; ovobjprint –a ―TopM Interface List‖ ―Selection Name‖=$a |grep –v ―TopM Interface‖ |grep –v ―OBJECT ID‖ | grep –v ―for all objects‖ done 266 23. Distributed NNM Mu ltip le cop ies of N N M can be set u p to share d ata and d istribu te p olling load s in a DIM configu ration. DIM is Distribu ted Internet Discovery and it is d escribed in the N N M Gu id e to Scalability and Distribu tion. A CS is an N N M Collection Station . An MS is an N N M Managem ent Station. Any cop y of N N M can be a CS, bu t not all versions of N N M can be an MS. Setting u p DIM requ ires m u ltip le licenses for the N N M p rod u ct. N ote that several im p ortant new er N N M featu res, like APA, m ay not be fu lly su p p orted in DIM environm ents. The APA in som e cases can elim inate the need for DIM. N ote also that if DIM is in u se on an MS and the APA is enabled , failover p olling w ill no longer w ork. If failover p olling is requ ired for a p articu lar architectu re on an MS, the APA cannot be enabled . D IM limitations DIM requ ires the u se of netmon-based d iscovery and p olling for the MS. APA p olling is su p p orted on CSs only. The Starter Ed ition and N N M 250-N od e Ed ition of N N M d o not su p p ort MS featu res. Extend ed Top ology view s only d iscover the m anaged , local and p rim ary top ology objects on a CS or MS; as of N N M 7.5 there is no integration of ET u nd er DIM. Generally, m ixed versions of N N M cannot p articip ate in DIM, bu t there are excep tions. Typ ically, sam e versions on varying p latform s are su p p orted . DIM can op erate across firew alls. The p orts that need to be op en are listed on p age 283 and there is m ore inform ation on this below . 267 Fognet’s Field Guide to OpenView NNM D IM overlap modes These m od es d efine the behavior of the relationship betw een CSs and MSs: allow Overlap : MS m anages 2 collection station s w ith overlapping d om ains. With thi m od e, there w ill be tw o entries in the topodb and one entry in the objdb on the MS. u nm anageSecond ary (u nm anage local): MS has ow n collection d omain; secondary objects unmanaged until failover . d eleteSecond ary (d elete local): MS d eletes CS copies of locally m anaged objects in topod b. netmon d iscovery is on. Ports used by MS and CS: This chart m ay not be accu rate for every version of N N M. Source Dest. Protocol SRC Ports DST Ports ------------------------------------------------------------------------------MS Mgd nod es UDP 1024-65535 161 SN MP Mgd nod es MS UDP 1024-65535 162 SN MP CS MS TCP 1024-65535 162 pmd MS Mgd nod es ICMP N/ A N/ A Any CS or MS TCP 8880 ovw w eb Any CS or MS TCP 8888 ovw w eb Any CS or MS TCP 9999 ovw w eb Any CS or MS TCP 3700 ovw w eb Polling through firew alls using D IM To configu re the p ort that nmdemandpoll op ens w hen listening for incom ing d em and p oll ou tp u t from a local or rem ote collection station, ed it the $OV_CON F\ nm d em and p oll.p orts file. Each line in the file m ay contain a single p ort nu m ber, a com m a-d elim ited list of p ort nu m bers, or an integer range. A valid integer range is of the form m -n, w here m and n are valid integers, and m is less than or equ al to n. Com m ents are d enoted by a nu m ber sign (#), and cau se the rem aind er of the line to be ignored . Blank lines are allow ed . Filtering N N M filters are u sed in several p laces for d iffering N N M fu nctions, and w ith N N M 7.0+, a second sep arate filtering facility is p rovid ed for ET fu nctions su ch as the APA Poller. ET top ology filtering is d iscu ssed sep arately on p age 82. 268 Distributed NNM Discovery filters lim it w hich objects the local topology d atabase contain s. Topology filters lim it w hich objects on a CS are passed to a MS. M ap filters lim it w hich objects are show n on user m aps. Failover filters specify the m ost critical nod es for w hich the MS takes over polling control should a CS becom e unreachable from the MS. Important N ode filters d eterm ine w hich nod es should never be flagged as second ary failures by netmon polling, and are d iscussed on page 58. APA Im portant nod e filtering uses a d ifferent m ethod ology as d escribed on page 82. Object filters d efine netmon-based d evices to w hich object-based polling applies. Persistence filters d efine objects that are placed in m em ory for backw ard com patibility w ith certain third party API applications by d isabling on d em and subm aps for those subm aps containing objects that pass the filter. DHCP filters id entify shared or floating IP ad d resses. ovfiltercheck, ovfiltertest and ovtopodump These com m and s can be u sed to trou bleshoot and verify filters: ovfiltercheck -v ovfiltertest ovtopodump –f <filt> Check filter syntax Prod uce the results of the filter and test against the topology database Test topology d atabase against filter V7.51 Interm ed iate Patch 18 introd u ced the “-x” op tion to the com m and ovtopodump. This op tion is sim ilar to ovtopodump –l, w hich p rints a su m m ary of d ata for the local nod e, only it p rints a su m m ary of d ata consolid ated from all collection stations. Examples of filter expressions H ere are som e com m only u sed exp ressions: CiscoDevices "Match all Cisco Routers" { "SNMP sysObjectID" ~ 1.3.6.1.4.1.9.* } DevicesWith3orMoreInterfaces ―‖ { numInterfaces > 2 } These exam p les show m ap filters that both exclu d e or inclu d e: 269 Fognet’s Field Guide to OpenView NNM Sets { EXCLUDEDNODELIST "Excluded Nodes" { "DAVE.acompany.int", "BILL.acompany.int" } INCLUDEDNODELIST "Included Nodes" {"JOHN.acompany.int", "FRED.acompany.int" } } Filters { NetsNSegs "All networks & segments" {isNetwork || isSegment} NTServer "sysObjectID for NT servers" { "SNMP sysObjectID" == ".1.3.6.1.4.1.311.1.1.3.1.2" } INCLUDEDNODES "Included Nodes" { "IP Hostname" IN INCLUDEDNODELIST } EXCLUDEDNODES "Excluded nodes" { "IP Hostname" IN EXCLUDEDNODELIST } } FilterExpressions { NtServerMap "NT Managed Systems" { !EXCLUDEDNODES && (NetsNSegs || NTServer || INCLUDEDNODES) } } Exclu sion filter for VLAN Interfaces. N ote u se of “| | ” vs. “&&”: NotVlan ―Not Vlan‖ isInterface && (("Selection Name" !~ "Vl") || ("Selection Name" !~ "VLAN")) } Using external files w ithin filters To help sep arate lists of sou rces from m aking the filters file grow too large, external files can be referred to w ithin the filters file. N ote that w hile w ild card s are su p p orted in filter d efinitions d irectly w ithin the filters file, they are not su p p orted in the external files. The exam p le below show s filters based on external files p op u lated w ith valid N N M nod e nam es or IP Ad d resses, one on each line: LRouters "Set of Routers from list" { c\:/cfg_files/routers } LTerminalServers "Set of Terminal Servers from list" { c\:/cfg_files/terminalservers } ListNodes "All nodes from list" { ( "IP Hostname" in LRouters ) || ( "IP Hostname" in LTerminalServers ) } Filtering idiosyncracies Filters u sing SN MP entities retu rn resu lts for the nod e entity and cannot be u sed to p rod u ce resu lts for interface entities. Any d efined m ap filters show u p in the H om ebase selection criteria. This can be a p ow erfu l tool for better cu stom izing d ynam ic view s. 270 Distributed NNM Examples of xnmtopoconf commands These are the DIM control com m and s: To ad d : To d elete: To check status: To test: xnmtopoconf xnmtopoconf xnmtopoconf xnmtopoconf -add collection_station -delete collection_station -print -test collection_station Remove secondary objects from the database When d isabling DIM, second ary objects can be rem oved from the d atabase u sing the follow ing com m and in ru nning N N M V7.51 w ith Interm ed iate Patch 18 or greater: ovtopofix –C -X D IM set-up in a nutshell On the collection station: Ed it / etc/ snm pd .conf to establish set com m unity string Ed it filters file to set up topology filter Ed it ovtopm d .lrf to ad d filter using LRF process (page 5), for exam ple: OVs_YES_START:pmd,ovwdb:O -f Routers: OVs_WELL_BEHAVED:15: On the m anagem ent station: Run: xnmtopoconf -print to see if CS is know n. If not, ad d using: xnmtopoconf -add -unmanage <CS> Options - Configure SN MP to set the SN MP set string for CS Test collection station : xnmtopoconf -test <CS> Configure failover: xnmtopoconf -failover <CS> Set status check interval: xnmtopoconf -interval [n] <CS> Manage CS: xnmtopoconf -manage <CS> Run: ovtopodump –RISC on the MS to monitor progress Migrating to non-D IM w ith APA Becau se of su bstantial im p rovem ents to scalability betw een version 6 and 7.5 on N N M, it m ay be d esirable to m igrate from m u ltip le N N M collection stations back to a single server architectu re. The ad vantages for d oing this are: few er N N M licenses, a single top ology to m aintain, better analysis across p olling bou nd aries by the APA, and be tter d ynam ic m ap s. Disad vantages are: a loss of failover ability if u sing DIM for failover, m ore com p lex and sensitive p erform ance tu ning tasks and non-d istribu ted p olling. N ote that if netmon-based p olling is 271 Fognet’s Field Guide to OpenView NNM not m anaging Level 2, there m ay be a large increase in the nu m ber of m anaged interfaces w hen sw itching to APA. Using WEB interface w ith an MS as Management Console The follow ing w hite p ap er in the $OV_DOC/ WhitePap er d irectory p rovid es som e tip s abou t configu ring w eb consoles to act as a “third tier” to the m anagem ent hierachry and thu s increase the nu m ber of op erators w ho can access a d istribu ted N N M installation: WebUIFrom RemoteConsoles.doc CS view s from MS Alarm Brow ser N N M V7.51 Interm ed iate Patch 18 introd u ced an enhancem ent to the ovalarmsrv p rocess that allow s som e Collection Station view s to be accessed via the Managem ent Station‟s Web Alarm brow ser by configu ring the follow ing file: $OV_CON F/ C/ xnmeventsExt.conf See the follow ing White Pap er for d etails on how to configu re this: $OV_DOC/ WhitePapers/ CrossLaunch.pd f 272 24. High Availability and Backup N N M p rovid es the ovbackup.ovpl scrip t to p erform backu p s of the N N M d atabases and im p ortant d ata. This scrip t calls the ovpause and ovresume com m and s that resp ectively qu iet and aw aken the d aem ons and force in-m em ory d ata to be w ritten ou t to the ap p rop riate files. Any op en ovw sessions w ill receive a p op u p inform ing them that OV d aem ons have been p au sed . The object, top ology, snm p Collect, and m ap d atabases u se N DBM flat file d atabases. These are “sp arse” files, and thu s there can be issu es w hen restoring if the backu p facility u sed isn‟t sp arse file-aw are. ovbackup.ovpl backs u p op erational and / or analytical files only. Base installation files like binaries and configu ration files are not backed u p by ovbackup.ovpl. Also, if ovbackup.ovpl is cu stom ized , be su re to save cop ies of the cu stom ized file since N N M u p grad es m ay over -w rite the file. To p iggyback onto the ovbackup.ovpl p rocess, w rite a scrip t that cop ies ad d itional d ata to be backed u p (as w ell as restored w ith ovrestore.ovpl) and p lace it in the follow ing d irectory: $OV_TMP/ ovbackup Automated database backups Data w arehou se backu p s occu r au tom atically by d efau lt. Daily transaction log consolid ation also occu rs by d efau lt. If ovbackup.ovpl is being u sed regu larly, then the DW is being backed u p tw ice. Often, errors arise if the au tom ated DW backu p and the ovbackup.ovplbased backu p s step on each other. To d isable au tom ated DW backu p s, m od ify the follow ing file and ad d ";" before the line that starts w ith "at": $OV_AN ALYSIS/ d efault/ solid .ini 273 Fognet’s Field Guide to OpenView NNM Then, rem ove old backu p s in : $OV_DB/ analysis/ d efault/ backup To p revent ovbackup.ovpl from backing u p the DW, ru n it w ith the – operational com m and line op tion. DW backu p files: Config file: Transaction logs: Backups: Error log: $OV_DB/ $OV_DB/ $OV_DB/ $OV_DB/ analysis/ analysis/ analysis/ analysis/ d efault/ d efault/ d efault/ d efault/ solid .ini log backup solerror.out Force backu p : ovbackup.ovpl -analytical -d dest_dir ovbackup and OpenView data protector (OmniBack) A best p ractice is to ru n: ovbackup.ovpl before ru nning Data Protector via cron or sched u ler. If ovbackup.ovpl is ru n as a p re-exec u sing Data Protector, a w rap p er scrip t is requ ired to establish the ap p rop riate environm ent variables. ovresume times out In higher scale d ep loym ents, the d efau lt tim eou t of 5 m inu tes can easily be reached , p articu larly w hen there are a lot of d ata collections, cau sing ovbackup.ovpl to fail. To increase the tim eou t, find the system call to ovresu m e w ithin the scrip t and ad d the “ -t” op tion to increase the tim eou t, for exam p le: system "$OV_BIN/ovresume -v -t900 > \ $OV_TMP/ovresume.output 2>&1"; Running ovbackup.ovpl as non-root user (UN IX) ovbackup.ovpl ru ns the ovpause and ovresume com m and s, w hich can only be ru n by root (by d efau lt). See the p roced u re on p age 12 for instru ctions on how to allow non -root u sers to execu te these com m and s. ovbackup.ovpl and D ata Warehouse interaction When the ovbackup.ovpl com m and is execu ted , it w ill instru ct the Em bed d ed Database server to initiate an online backu p . In ord er to allow a roll-forw ard recovery, the d efau lt backu p sched u le m u st be d eactivated . The p rocess for d oing this is: 274 High Availability and Backup 1. 2. 3. Copy $OV_DB/ analysis/ d efault/ solid .ini file to $OV_DB/ analysis/ d efault/ solid .ini.old file. Ed it $OV_DB/ analysis/ d efault/ solid .ini file. Com m ent out "At=<tim e> backup" entry, by inserting a ";" at the beginning of the line. For exam ple: ;At=01:00 backup Save $OV_DB/ analysis/ d efau lt/ solid .ini file . 4. 5. The em bed d ed d atabase w ill now be backed u p only w hen the ovbacku p .ovp l com m and is ru n. If u se of ovbacku p .ovp l is stop p ed in the fu tu re, the d efau lt backu p shou ld be restored by cop ying the solid .ini.old file back to solid .ini. ov dbcheck daemon not starting (MS SQL) On Wind ow s system after a reboot, if ovdbcheck fails to start it m ay be becau se the SQL server is still recovering from the reboot and is not available to N N M yet. Sim p ly w aiting for the DB to recover shou ld su ffice. To avoid this issu e, alw ays try to stop N N M w ith the ovstop com m and p rior to rebooting the server. Backing up SN MP configuration data The com m and to save SN MP configu ration d atabase, w hich contains p olling and SN MP com m u nity nam e settings is: xnmsnmpconf -export <outputfile> Integrating N N M w ith OVO database backup Use this w hen ru nning N N M on a server along w ith OVO (O p enView Op erations). The d atabase backu p integration scrip ts are in $OV_CON F/ ovbacku p / checkp oint/ op erational d irectory, and contain both nnm_checkpoint.ovpl and ito_checkpoint.sh and $OV_CONF/ovbackup/pre-pause (w hich contains ito_oracle.sh). Backing up the Solid database Solid DB grow s very large in highly-scaled environm ents u sing ET. A Solid DB greater than 2 gigabytes in size is not u nu su al. The Solid d atabase resid es in: $OV_DB/ analysis/ d efault By d efau lt, the Solid backu p p rogram is au tom atically lau nched d aily at 11p m . So if ovbackup.ovpl is in u se, then the Solid d atabase is u neccessarily backed u p tw ice. The au tom atic backu p is m ad e to: 275 Fognet’s Field Guide to OpenView NNM $OV_DB/ analysis/ d efault/ backup To d isable the au tom atic Solid backu p , p u t a sem icolon before the follow ing line in the $OV_DB/ analysis/ d efau lt/ solid ini.file then restart the Op enView d aem ons: At=23:00 backup To d isable Solid backu p w hen ru nning ovbackup.ovpl, ru n instead : ovbackup.ovpl –operational To change the location of the Solid d atabase backu p , sp ecify a new p ath in the solid .ini file that is relative to $OV_DB/ analysis/ d efau lt, for exam p le: [General] BackupDirectory=../ ../ ../ ../ ../ ../ tm p/ solid bak Map exports and imports The ovw GUI Map Exp ort/ Im p ort facility p reserves IPMAP cu stom izations m ad e for a sp ecific top ology. Legacy d atabases requ ire occasional rebu ild ing, p articu larly in d ynam ic environm ents . Most N N M ad m inistrators create cu stom izations to their m ap s u sing “containerized ” realm s and cu t-and -p aste op erations. These cu stom izations can be p reserved u sing the Map Exp ort featu re. When restoring a m ap u sing an exp orted m ap file, the assu m p tion is that the u nd erlying top ology d ata in the d atabases is essentially the sam e. If it is not, m any objects m ay be p laced in the “N ew Object H old ing Area.” Before im p orting a m ap , first m ake su re that the d iscovery is com p lete and that any objects that w ere m anu ally p op u lated into the d atabases via loadhosts are in p lace before p erform ing the m ap im p ort. Map snapshots A m ap snap shot, w hich is also p erform ed from the ovw GUI, cap tu res a cop y of the entire m ap d atabase, and allow s for the “freezing” of IPMAP top ology statu s at a p articu lar p oint in tim e. Use snap shots only for that p u rp ose: to cap tu re the statu s of the top ology at that p articu lar m om ent in tim e. Map snap shots can also be issu ed externally u sing the ovmapsnap com m and . 276 High Availability and Backup ovtopodbsnapshot.ovpl This su p p ort tool w as introd u ced in N N M V7.5 and it is u sefu l for creating backu p s and restoration p oints for both N N M‟s ovw and ET top ology d atabases and associated configu ration files. N ote that it issu es a com p lete N N M d aem on shu td ow n (ovstop) w hen it is execu ted : $OV_SUPOPORT/NM/ovtopodbsnapshot –c <outputfile> This tool can also be u sed effectively to to m aintain a hot-stand by server. N ative backup and restore commands (UN IX) Use these w hen choosing not to u se the ovbackup.ovpl scrip t. N ote N N M legacy d atabases are “sp arse” files, so m u st be backed u p w ith sp arse file-aw are tools. For H P-UX, u se fbackup/frestore and for Solaris, u se ufsdump/ufsrestore. cpio is also sp arse file-aw are. Backup sparse file d atabases u sing cpio: cd $OV_DB/openview ; find . | cpio –pdmux \ /dir/backupfile >> /dir/logfile 2 >> /dir/logfile Backup and restore sparse file DB‟s using fbackup or frecover to or from tape: fbackup –oi $OV_DB/openview –f /dir/backupfile frecover –x –v –o –f –s /dev/rmt/om \ -i $OV_DB/openview D IM Configuration for a hot standby system The follow ing im p lem entation of DIM as a hot stand by u ses the coop erative-ind ep end ent DIM m od el d escribed in the gu id e to Distribu tion and Scalability for N N M. An ad d itional license is requ ired , bu t in m ost cases, a d iscou nt can be negotiated throu gh the sales p rocess. The tw o N N M servers shou ld have the sam e visibility to the netw ork. Generally, the hot stand by shou ld be ru nning the sa m e OS versions and have a sim ilar hard w are configu ration as the p rim ary system . First, m ake su re both stations have d iscovered the sam e top ology. Cop y the configu ration files of interest from the p rim ary to the stand by server w hen configu ring the stand by for the first tim e. Of p articu lar interest w ill be any netmon seed files that m ay have been 277 Fognet’s Field Guide to OpenView NNM configu red and the netm on.noDiscover file. Also, cop y the trap d .conf file for the p rim ary. Create the top ology filters as follow s: For Prim ary N N M server nam ed “Prim ary,” create filters: Primary "" { "IP Address" ~ "1.2.3.4" } UnManaged "" { "IP Status" ~ "Unmanaged" } For the sam e p rim ary server, create FilterExp ression : Topofilter ""{ !Primary && !UnManaged } Create the sam e filters on the stand by server, su bstitu ting the nam e of the stand by server w here it says “Prim ary” above. Up d ate ovtopmd LRF files to call the new top ology filters on each server, p er the d etailed p roced u re in the Scalability and Distribu tion gu id e. On the p rim ary server, ru n: xnmtopoconf -add -over AllowOverlap ServerB xnmtopoconf -failover ServerB Set the statu s check interval on the p rim ary server u sing: xnmtopoconf -interval <n> <node> <n> is nu m ber of m inu tes betw een statu s checks betw een the m anager and the collection station, and <node> is the nam e of the stand by server. It m ay take a few hou rs for ovrepld to stabilize the tw o top ologies. N on-D IM configuration for a “hot standby” A hot stand by can be configu red u sing the follow ing ou tline. An ad d itional license is requ ired , bu t in m ost cases, a d iscou nt can be negotiated throu gh the sales p rocess. Generally, the hot stand by shou ld be ru nning the sam e OS versions and a sim ilar hard w are configu ration. The tw o servers m u st be ru nning the sam e version of N N M at the sam e p atch level. The tw o N N M servers shou ld have the sam e visibility to the netw ork. There are actu ally several w ays this can be accom p lished , and this m ethod is neither the best nor the m ost efficient. 278 High Availability and Backup Cop y configu ration files of interest from the p rim ary to the stand by server w hen configu ring the stand by for the first tim e. Of p articu lar interest w ill be any netmon seed files that m ay have been configu red and the netm on.noDiscover file. Also, cop y the trap d .conf file from the p rim ary. On the p rim ary server, exp ort the SN MP configu ration d atabase and cop y to the stand by. Im p ort it on the stand by u sing: $OV_BIN/xnmsnmpconf –import <file> Allow the stand by server to d iscover the entire netw ork. In d oing so, the stand by server w ill set itself as a trap recip ient on the m anaged nod es. Com p are the resu lts of this server‟s d iscovery to that of the p rim ary. Differences m ay be attribu table to netw ork visibility if the p rim ary and stand by are on d ifferent segm ents. Run: ovbackup.ovpl on prim ary N N M server Copy ovbackup.ovpl results to stand by server. Run: ovstop the run ovrestore.ovpl on stand by server. Populate the stand by server‟s FQDN in: $OV_DB/ openview/ ovw db/ ovserver In essence, the id ea is to let d iscovery p olling ru n on the stand by server, bu t not statu s p olling. To d o this, ru n: xnmpolling –statPollOff When u sing APA p olling instead of netmon p olling: ovdelobj $OV_LRF/ovet_poll.lrf; ovstart ovet_poll To enable the stand by: xnmpolling –statPollOn or: ovaddobj $OV_LRF/ovet_poll ovstart ovet_poll The above step s p rovid e an ou tline for getting started , and they can be inclu d ed in a nightly or w eekly set of scrip ts to keep the server‟s synchronized . Rem em ber, that w hen sw itching from stand by back to p rim ary, the p roced u re shou ld be reversed so that any changes that occu rred w hile the p rim ary w as d ow n are not lost w hen the p rim ary is brou ght back online. 279 Fognet’s Field Guide to OpenView NNM N N M as a highly available service N N M as a highly available ap p lication in UN IX clu sters has enjoyed increasing su p p ort from H P, bu t w ith p roblem atic su p p ort in earlier versions. After N N M V7.01, clu stering is su p p orted for H P‟s MC ServiceGu ard 10.06 or greater, Su n Clu ster 2.2 or greater, and Veritas Clu ster 2.0 or greater. N N M‟s ad d ed su p p ort in V7.01 for m igratable IP ad d resses has enhanced N N M‟s clu sterability. V7.51 Interm ed iate Patch 18 ad d ed su p p ort for Wind ow s Clu sters, and a w hite p ap er d escribing its configu ra tion in the V7.53 $OV_DOC/ WhitePap ers d irectory. N N M 6.2 (p atched ) and higher su p p orts MC ServiceGu ard and Veritas Clu ster, bu t floating IP ad d resses m u st be u nd iscovered by p lacing them in the netm on.noDiscover file. N N M can read the ServiceGu ard MIBs in the cmsnmpd agent, w hich allow s N N M to sep arate floating IPs, bu t if this agent stop s ru nning, instability cou ld resu lt. N ote that clu stered solu tions have very sp ecific hard w are requ irem ents. MC ServiceGu ard , for exam p le, w on‟t ru n on m ost of H P‟s w orkstation class hard w are. Sim ilarly for SUN hard w are, there are m inim u m requ irem ents for high -sp eed channel connects. The $OV_CON F/ ov.conf file contains configu ration settings that p ertain to u sing N N M in a clu ster. It has its ow n m an/ ref p age. Ad d itional N N M licenses are requ ired to ru n N N M on clu ster m em ber nod es, bu t these licenses shou ld be available at su bstantial d iscou nts. The choice as to w hether to achieve high availability via clu stering technology or via im p lem enting a DIM solu tion is a hard one, and alm ost alw ays com es d ow n m aintainability issu es rather than cost or com p lexity issu es. A general ru le of thu m b is that if the staff is fam iliar and com fortable w ith clu stered environm ents, clu stering is the p referred m ethod . Likew ise, DIM shou ld be the choice for a DIMcom p etent staff. See p age 277 for an exam p le DIM configu ration that p rovid es a highly-available N N M configu ration. Increasing favor tow ard s d isaster recovery architectu res su ggest DIMbased solu tions, w hich lend them selves m ore easily to d isaster recovery architectu res becau se clu stering solu tions can be p roblem atic over great d istances. 280 High Availability and Backup Syslog in an HA environment If the syslog facility is to be u sed in an H A environ m ent, then syslogTrap requ ires being installed on all nod es in a H A clu ster. This is d u e to u p d ating files that are not u nd er the shared d isks. Ad d itionally, it is necessary to stop syslogTrap w hen bringing d ow n the N N M p ackage. To d o this, it is recom m end ed that the follow ing com m and s be ad d ed to the clu ster ru n com m and s and halt com m and s: Run com mand s: / opt/ OV/ bin/ OpC/ opcagt –start H alt com m and s: / opt/ OV/ bin/ OpC/ opcagt -stop / opt/ OV/ bin/ OpC/ opcagt –kill The ord er betw een op cagt and ovstart/ ovstop calls d oes not m atter. N N M and multiple N IC cards H ighly-scaled N N M installations can benefit from u sing m u ltip le N ICS on the m anagem ent server, bu t this op tion has strict lim its, m ostly d u e to OS p rocessing requ irem ents w ithin the netw ork stack. N N M‟s d efau lt p oller netmon is single-thread ed , so ou tbou nd p olls are likely to be bou nd to the sam e interface anyw ay. N N M 7.01 introd u ced the APA p oller, w hich is m u lti-thread ed ; this p oller can take ad vantage of m u ltip le interfaces for issu ing p olls and receiving rep lies. For red u nd ancy, extra N ICs m ay be set u p to act as stand bys. In this case, set u p the backu p N IC(s) w ith the sam e d efau lt rou tes as the p rim ary N IC. Set the rou te m etric for the backu p N ICs higher than the p rim ary, and the rou ter shou ld sim p ly u se the backu p if the p rim ary fails or becom es too congested . Also, the USE_LOOPBACK setting in the $OV_CON F/ ov.conf file w ill stop N N M from bind ing to a sp ecific N IC. Managing migratable IP addresses Clu ster virtu al IP ad d resses that d o not fall u nd er N N M‟s ability to hand le H SRP (see p age 245) can be hand led u sing the netm on.m igratable configu ration file. For m ore inform ation, see the m an/ ref p age for netm on.m igratable, w hich w as introd u ced in V6.31. When netmon find s that an interface flagged as m igratable ap p ears on a new d evice, it d oes not attem p t to m erge the nod es. Instead , it allow s m ovem ent of the interface to the new d evice. 281 Fognet’s Field Guide to OpenView NNM The IP ad d ress belonging to a m igratable interface is given a low er p riority for u se w ithin the nod e nam ing algorithm . More inform ation on the ord er that is u sed to d eterm ine nod e nam es can be fou nd on p age 24. The configu ration file accep ts single IP ad d resses, IP ad d ress ranges, or IP ad d ress w ild card s. The file is read u p on netmon d aem on startu p , bu t a re-read of the configu ration file is forced by ru nning: xnmpolling –event Trou bleshoot issu es stem m ing from netm on.m igratable by u sing the netmon –a 115 tracem ask. See p age 67 for d etails on netmon tracing. 282 25. Firewalls and Security Many issu es arise w hen attem p ting to m anage nod es throu gh firew alls. There is a w hitep ap er that ship s w ith N N M on firew all d etails, bu t it is p ossible this is ou t of d ate w ith the m ost recent versions of N N M. N N M Ports used Many of these p orts are only accessed via the local loop back of the N N M server. Used by N N M d aem ons: SN MP requests (netmon, APA): source port ud p 1024-65535 d est port ud p 161 SN MP replies (netmon, APA): source port ud p 161 d est port ud p 162 SN MP traps (ovtrapd): d est port ud p 1024-6553 source port ud p 162 ICMP requests and responses (netmon, APA) Used by N N M Web interface: 283 ovalarmsrv : source port tcp 2345 (V6.1-) 2953 (V6.2+) source port tcp 2346 (V6.1 2954 (V6.2+) ovhttpd : source port tcp 8880 (V6.2- UNIX) source port tcp 3443 (V6.31+ UN IX) source port tcp 80 (Window s) ovas : source port tcp 7510 (V6.31+) ; 8005 (V7.01+) ovwdb : source port tcp 9999 (V6.1-) 2447 (V6.2+) source port tcp 37XX, one for each jovw session, sequentially from port 3700 Fognet’s Field Guide to OpenView NNM Reconfiguring ports used by N N M The p orts u sed by m ost d aem ons can be configu red via LRF sw itches. See p age 5 for how to m od ify LRF files. Som e p orts w hich certain d aem ons listen on, how ever, cannot be reconfigu red . N N M‟s ovtrapd m u st listen on UDP p ort 162 and this cannot be changed . The p ort it send trap s from (UDP 161), how ever, can be changed . N ote that even if the trap receiver p ort cou ld be changed , every SN MP agent in the enterp rise w ou ld also have to be reconfigu red to send trap s on a d ifferent p ort, and it is an u nlikely feat to accom p lish for all agents. pmd u ses TCP (not UDP) p ort 162 for talking to other cop ies of N N M, and this p ort ap p arently cannot be changed . This is an issu e, thou gh, only if ovrepld is ru nning, w hich it d oes only w hen u sing DIM (see p age 267) or N N M-to-N N M event forw ard ing. Highly-secure netw ork management scenario Most netw ork m anagers w ou ld consid er the follow ing recom m end ations extrem e, bu t som e u nu su al or ou t-of-control environm ents call for extrem e m easu res to su ccessfu lly and secu rely m anage them . The follow ing scenario p rovid es su ch an exam p le: Create a separate VLAN or managem ent network for the LAN infrastructure. Protect this m anagem ent network from the rest of the network via a firew all or at a m inim um using access-lists. Use d evice-based access lists or sim ilar technology to lim it SN MP access to the WAN infrastructure from the m anagem ent netw ork. Sim ilarly lim it access to the network m anagem ent servers. Consid er the use of SN MPv3 on particularly critical and vulner able d evices. Finally, alw ays m ake sure the appropriate version of cod e and or patches is running in the m anaged environm ent. N N M ICMP polls versus ping sw eep attacks Som e firew alls rep ort p ing sw eep attacks from the N N M server. N N M-issu ed ICMP p olls contain a string of all zeros in the p ayload , and the p acket is alw ays 64 bytes. This is an u nu su al p acket configu ration and cannot be re-configu red . If the “-b” op tion is set in the netm on.lrf file, this m ay be the sou rce of u nu su ally heavy ICMP traffic. See the netmon m an/ ref for d etails. 284 Firewalls Security issue w ith ICMP ping (UN IX only) Som e IT secu rity d ep artm ents exp ress concern abou t N N M‟s statu s p olls. N N M u ses root to execu te p ing. This is a requ irem ent as only root u ser can op en raw sockets on UN IX. The raw socket is only op en for a short tim e and only to send and receive 64 bytes of d ata. It only accep ts the ICMP echo resp onse p acket com ing back. D iscovering into subnets using N AT N N M read s a target‟s SN MP ip .ip Ad d rTable and m ay rep ort ad d resses that p rod u ce confu sing top ology view s and rep ort d ow n statu s since they are only reachable w ithin the d iscovered su bnet, and not from the N N M server. One w ay to avoid this is to block the visibility of the offend ing IP ad d ress tables u sing SN MP cu t view s. If seeing the non-N AT ad d resses is d esirable, som etim es these ad d resses are treated as second ary an d not rep orted in the IP Ad d ress table, then N N M ignores them by d efau lt. In this case, the “S” flag in the oid _to_typ e or H Poid 2typ e files can be set to force read ing of second ary ad d resses. See p age 7 for m ore on m od ifying these files. H ere is an exam p le set of Cisco IOS com m and s to allow N N M to view N AT and to exclu d e ip .ip Ad d rTable: snmp-server view NAT-view mgmt included snmp-server view NAT-view enterprises included snmp-server view NAT-view ip.20 excluded snmp-server community whatever view NAT-view rw 1 Monitoring D MZ devices Som etim es, it is not d esirable to loosen DMZ -intranet firew all ru les to allow netw ork m onitoring insid e a DMZ. One alternative is to set u p a single host w ithin the DMZ to act as an SN MP p roxy. ICMP-based p olling w ou ld be p reclu d ed , bu t if all the DMZ hosts su p p ort SN MP, either netmon’s netm on.snm p Statu s file can be u sed (see p age 57), or ET top ology filters in conju nction w ith APA p olling can be u sed to control w hat d evices get p olled (see p age 82). In environments where there is a large number of hosts in the DMZ, setting up another copy of NNM in the DMZ and using DIM (see page 268) is also a workable solution. Some third party products, such as Tavve‟s eprobe (w w w .tavve.com ), are specifically designed to gather network management data and 285 Fognet’s Field Guide to OpenView NNM securely communicate to an internal network management server. Extended Topology configuration GUI passw ord The Extend ed Top ology configu ration GUI p assw ord is in cleartext in: $OV_AS\ w ebapps\ topology\ WEB-IN F\ d ynam icView sUsers.xm l 286 26. Product Details This section su m m arizes new N N M version release n otes, d ifferences betw een N N M p rod u ct bu nd les, OS-based fu nctionality d ifferences and a su m m ary of m ajor IT m anagem ent p rod u cts in the Op enView su ite. N N M 8i N N M 8i (V8.00) w as released in N ovem ber, 2007 and is a com p letely d ifferent p rod u ct that any p reviou s version of N N M. Up u ntil this version, virtu ally every featu re ever introd u ced to N N M since version 3 has rem ained w ith the p rod u ct for backw ard com p atibility. That changes d ram atically w ith N N M 8i. The au thor estim ates that 85% of the cod e in N N M 8i is brand new . There is no u p grad e p ath from p reviou s versions of N N M; it m u st be installed on a fresh system . H P p lans to su p p ort and release new versions of N N M 7.x in p arallel w ith N N M 8.x for at least the w hole of 2008. Migration tools for som e legacy N N M cu stom izations and som e d atabase d ata w ill be available w ith N N M 8.10, d u e in the fall of 2008. Most fam iliar N N M featu res and p rod u ct elem ents are totally absent from 8i, inclu d ing the ovw GUI, all xnm ap p lications and all the d atabases. N N M 8i is a 64-bit ap p lication that requ ires 64-bit hard w are ru nning 64-bit OS versions and is bu ilt on a J2EE ap p lication server. It consists of fu nctional su bsystem s in an n-tier architectu re. The m od u les are organized arou nd a central notification bu s that p rovid es for com m u nications betw een m od u les and , in the fu tu re, com m u nications betw een d istribu ted cop ies of N N M 8.x. The only m ajor su bsystem that w as p orted over from old er cod e is the Extend ed Top ology cod e, bu t w ith a d ifferent d atabase, a d ifferent GUI, and d ifferent configu ration entry p oints. The pmd event su bsystem rem ains in p lace, bu t only to act as a forw ard er or receivers of N N M events to/ from legacy cop ies of N N M. The new GUI can also connect to legacy installations of N N M and d isp lay those p rod u ct 287 Fognet’s Field Guide to OpenView NNM elem ents that w ere w eb-based . One cu riou s artifact from legacy N N M is the ovspmd p rocess and the fam iliar ARF p rocess (see p age 6). The au thor p rep ared a p resentation on N N M 8i for the 2008 H P Softw are Universe conference in Las Vegas, bu t the p resentation w as rejected by H P and not p resented there. It w as p resented for som e Vivit local chap ter u ser grou p m eetings, how ever. It can be d ow nload ed from : w w w.fognet.com / hpsu08.pd f N N M 8i architecture Use the d iagram below to com p are and contrast 8i‟s architectu re w ith p reviou s versions. N ote that it‟s com p letely d ifferent from the architectu re d ep icted on p age 69. 288 Product Details 289 Fognet’s Field Guide to OpenView NNM Product feature deltas by N N M version 8.01 (Release Date: 1/ 17/ 08): AKA N N M 8i. See section on N N M 8i above. 8.00 (Release Date: 11/ 26/ 07): AKA N N M 8i. See section on N N M 8i above. 7.53 (Release Date: 4/ 03/ 08): Ad vanced Ed ition only: Container View Access Control N ode Status and Interface Status View s Alarm View Use ET data in Path View Connect end nod es from another ET zone Interface configuration change d etection in APA Im port xnm snm pconf tim eouts & retries into ET ET com m unity string d iscovery Port ad m in tool in Dynam ic View s Stand ard Ed ition & Ad van ced Ed ition: Linux platform support (See OS section below ) H P-UX 11iv3 support (See OS section below ) Solaris Containers/ virtualization support VMw are ESX3.0 support Wind ow s Cluster support JRE 1.5 support 7.52 (Release Date: ?/ 07): This w as a lim ited release for sp ecial cu stom ers asking for extend Linu x p latform su p p ort inclu d ing Red hat Linu x 2.6 7.51 (Release Date: 8/ 24/ 06): Ad vanced Ed ition only: Container View s – ovw-like view s for ET Interface View s Interface Discovery Filtering Increm ental N od e Discovery for ET Cisco Discovery Configuration Connection Ed itor Enhancem ents APA Im provem ents Im proved ET Disvovery perform ance Topology API stability im provements Contrib App: SmartPath Stand ard Ed ition & Ad vanced Ed ition: N N M & OVPI integration pack Automatic m erging of user-configurable files d uring patch updates Default num ber of hops in Neighbor view is now configurable Support for Std & Ent Ed itions of Wind ow s 2003 R2 Support for Tom cat v5 Support for Apache v2 Support for Solid DB v4.5 7.5 assorted p atch release featu res: Ad vanced Ed ition only: APA param eters add ed to load only polled objects into m emory 290 Product Details APA param eters add ed for backoff polling for particularl d evices APA preferred IP m anagem ent add ress param ters ad d ed Im proved SN MP NoSuchObject hand lin g in APA ET-based filters for Nod e View add ed VRRP View Expand ed ET d evice support Stand ard Ed ition & Ad vanced Ed ition: Java Web Start for w ind ow s-based Dynam ic View s clients IPMAP Filter of netw ork sym bols connected to a gateway ovtrapd can block traps based on com bo of IP ad d r‟s and event OID‟s ovtrapd automatically unblocks blocked d evices after trap storm ovtrapd .conf sw itch for rate of event flow netmon sw itch to d isable d efault Anycast functionaility netmon sw itch to d isable BGP routing tables queries 7.5 (Release Date: 8/ 12/ 04): Ad vanced Ed ition only: Cisco Card support show s “board” entity d etails Trunk Support im provem ents w ith new icons OAD Im provem ents: VLAN , H SRP, Pathview , Telnet Rem oval of ovet_auth and ovet_d ffile Im proved Topology Filtering Im proved Layer Three connectivity in d ynam ic view s N ew APA Status for Dynam ic View s: “Not Monitored” N ew Topology Report for ET: Not SN MP Supported N eighbor View context-sensitive VLAN View launch Im provem ents to syslog tem plate Support for MS SQL (N T Version) Stand ard Ed ition & Ad vanced Ed ition: View alarm s for selected nod es in d ynam ic view s Support for Window s 2003 (N T Version) Upd ated Java Brow sers, Plug-Ins, and Virtual Machines Display layer 3 ed ge connectivity across WAN s ovtrapd .conf suppresses trap storm s and traps from ad d r‟s 7.01 (Release Date: 1/ 14/ 04): AE Only: APA Poller can take over all netmon status polling Fixes 7.0 issues w ith OID_to_sym Fixes 7.0 issues w ith d eleting an OAD SE Only: Support for H P-UX on Itanium w ith H P-UX 11.23 Support for Red H at Linux Ad vanced Server 2.1 7.0 (Release Date: 10/ 8/ 03): SE/ AE Prod u ct Stru ctu re introd u ced AE Only: Introd uction of syslog parser (AE UN IX versions only) Extend ed Topology features supported on N T version OAD Support introd uced 291 Fognet’s Field Guide to OpenView NNM APA introd uced only for H SRP and OAD polling (Active Problem Analyzer) Support for filters in brid ge.noDiscover file ET d iscovery status m oved to H om e Base ovet_topod um p.ovpl introd uced ET Discovery im provem ents (VLAN , autozone, etc) SE & AE: Im proved d ynam ic view s: Introd uction of Hom e Base, m ulti-fram ed view s, “active tables,” Authorization, authentication manage nod es via Dynam icView sUsers.xm l, better scalability, many new icons, extensible m enus, poster printing, toggle port labels, nod e, interface d etails pages. N ew event correlations introd uced and updated Mapping of SN MP sysObjectID to sym bol m oved from $OV_CON F/ oid _to_sym to $OV_CON F/ oid_to_sym_reg/ Support for 31 bit subnet m asks netm on.m igratable introd uced netm on.MACnoDiscover introd uced SE Only: Support for Red H at Linux Ad vanced Server 2.1 6.41 (ET 2.01) (Release Date: 3/ 12/ 03): Correlation Com poser Introd uced N ew correlations, includ ing DeDup and d ed up.conf Cisco H SRP d iscovery, m onitoring, and 3 new H SRP View s IPV6 d iscovery & m onitoring, includ ing 4 new IPV6 view s Im proved Dynam ic View s: Dynam ic View s m enu bar, N eighbor View , Sum mary View , Print, Icon Label Control, Im proved zoom ing, signed applets, Discovery/ d evice support for MPLS (isMPLS), IPV6 (isIPV6), BGP4 (isBGP4), H SRP (isH SRP), OSPF (isOSPF), STP (isSTP), VRRP (isVRRP), Wireless (isWireless), ET introd uces Zoning and conversion to Solid Database Data w arehouse support for Oracle 9 AutoPass Licensing introd uced Jakarta Tom cat ($OV_AS) replaces $OV_H PAS (App Server) JRE 1.4.1 support in $OV_JRE Data Collector im provem ents and new (suspend ed ) collections N N M and OVPI integration (Perform ance Insight) 6.31 (ET 1.51) (Release Date: 6/ 19/ 02): Web server port m oved from 8880 to 3443 (UNIX versions) Major Event Correlation ch anges (N odeIf) affecting status netmon status event change: nod e-centric to interface-centric Web Based alarm brow ser support for xnm eventsExt.conf Dynam ic View s introd uced w ith Stations, Internet, N etwork, Segm ent, Path, N eighbor and Nod e View s JRE 1.3.1 support in $OV_JRE, 1.4 on N T Connected N od es renam ed Port-Ad d ress Mapping 6.2 (Release Date: 4/ 25/ 01): 292 Product Details N ode View introd uced using Java Plug-in V 1.2 Cisco CDP View introd uced Port Labeling Introd uced via Show Connection Labels Show Path and ovtopod um p .ovpl SN MP and object-based status polling by netmon connectorL2Ports, nonConnectorL2Ports netmon sw itches Discover/ d evice support for isFram eRelay, isRMON , isRMON2, isCDP, isATM, isDS1, isDS3, isSON ET. Display Brid ge MIB table configur ation m enu bar item View Connected Nod es m enu item sorted by port, VLAN netm on.equivPorts introd uced for im proved port aggregation Attribute-based Data Collection Using Filters Baseline Threshold Setting using Standard Deviation based on the data collected and stored in the data w arehouse. See statTim eRanges.conf(4) Collection of Ping Response Tim e and Ping Retries Data collection throughput rate has im proved more than 10x Out-of-the-box data collections configured Im proved support for d evices that chan ge their SN MP instance to interface mappings after a reconfiguration. SN M PCollect performance via ovstatus -v snm pCollect. ovdw query -secureFile and -force options N ew Reports - Top ICMP Ping Response Tim e, Top ICMP Ping Retries, Top RMON Segm ents By Octets, Top Fram e Relay Congestion Automatic Report Generation - Availability, and Inventory Trend data exports and trim are now enabled by d efault N ew , faster ECS 3.1 engine - Circuits im proved , faster Zoom to H ighlight - H ighlight nod e using xnm events or Find Ed it:Find is available in the Java-based N etwork Presenter. Menu Graying in N etwork Presenter N ew telnet, traceroute, and RMON m enu and pop -up item s Link Managem ent Menu Item s in ovw :ATM, Fram e Relay, DS1/ E1 Serial Line, DS3/ E3 Serial Line, SONET/ SDH, along w ith various Perform ance, and Fault menu item s. Tools:Unused IP Ad d resses. OV_EventStorm event if an event storm is d etected . N am e Service Response Tim e events: OV_DN S_Perform Err, OV_DN S_Perform Warn Data Collector (snm pCollect) response tim e event: OV_DataColl_Busy netmon perform ance via ovstatus -v netmon ovactiond Trusted Com m and s in trusted Cm d s.conf $OV_CON F/ trusted Cm d s.conf/ ALLOW_ALL file SN MP Com m unity String Discovery via netm on.cm str N ew ovw d b field : Preferred SN MP Ad d ress Firew all Support of Traps Received Through a N AT d evice via -u option in ov_event ovdumpevents -t and -l options for tail and for tim e w ind ow M Flag in H Poid 2type File for m ulti-hom ed nod es Manuals Available as .pd f Files ovwdb TCP port num ber change from 9999 to 2447 ovalarmsrv ,ovalarm srv_cm d : 2345 to 2953, 2346 to 2954 Obsolescence of Excel report tem plates 293 Fognet’s Field Guide to OpenView NNM Wind ow s-only Featu res: Installation into d irectories w ith spaces now supported . Default d ir C:\ Program Files\ H P OpenView \ NN M\ . Drag-and -Drop from ovw map. Microsoft Term inal Server on Wind ow s 2000. Tip-of-the-d ay. Use of Winsock-2 instead of Winsock-1 libraries. OVSH ELL and OVH IDESHELL trapd .conf Keyw ord s N N M 6.1 (Release Date: 10/ 29/ 99): Web Based Reporting Service via JDK 1.1 “SegRed ux” - Im proved Layout for Sw itched Environm ents N T Support for Window s® 2000 Support of secure w eb servers (https:/ / ) Microsoft Managem ent Console (MMC) Snap -in Im proved integration w ith H P OpenView ManageX Software Update Menu via H elp:N N M->Patches N N M 6.01 (Release Date: 4/ 5/ 99): N ew capability in xnm graph to automatically configure d ata collections for the data currently being show n on the graph via File >Configure Data Collector... m enu item . xnmgraph and rnetstat support invocation for a specific interface (not just a nod e). N ew rnetstat options for nod e or interface ovexprguru introd uced to graph statistics based on the configuration of the particular d evice being graphed Som e new expressions are shipped as part of m ibExpr.conf N N M 6.0 (Release Date: 10/ 27/ 98): Event Correlation Services (ECS) Support introd uced Data w arehouse introd uced Lightw eight em bed d ed database for NN M (Solid ) Support for relational d atabases via ODBC ovtopmd -Ro/-Ri options (Oracle/ Ingres) obsoleted Support for exp ort of topology (inventory) data Support for export of snmpCollect (trend ) data Support for export of alarm (events) d ata Excel tem plates to retrieve d ata into Excel spread sheets Im proved Alarm Brow ser Support for relational event d isplay of correlated alarm s Support for m ultiple operators - when one operator an All operators see acknow led ges/ d eletes/ reclassifies N ew ind icator for correlated events event brow ser (xnm events) renamed 'alarm brow ser' Menu item s have changed from 'event' to 'alarm '. The Binary Event Store (BES) replaces the trapd .log file H P OpenView Java-based Web Interface introd uced : H P OpenView Launcher, N etw ork Presenter, Web alarm brow ser, SN MP Data Presenter ovbackup.ovpl introd uced w ith ovpause, ovresume Sim pler SPU-keyed licensing rep laces Nod e locks (netls) Rem ote Console support across platform s 294 Product Details Collection Station support via ovrepld across platform s Automatic managem ent station/ Collection Station failover DHCP support DynaText Brow ser Data Collector Enhancem ents perform ance im provem ents via -I snmpCollect option Multiple sysObjectId s per collection Multiple nod e/ instance combinations per collection Alarm generated when nod e up but SN MP request fails Back-off d eferrals of collections w ithin the first hour Menu Item s re-organized for platform com patibility Repeater MIB (RFC 2108) support SN MPv2C support in all NN M applications Im proved status polling perform ance (connector d ow n ECS) N ew "Map ->Export..." and "Map ->Im port..." menu item s Registration files support platform -specific d efs (#ifd ef) Integration and auto-d etection of H POV ManageX prod uct Installation im provem ents Perl 5.003 automatically installed Apache Web server automatically installed (UN IX only) Java Runtim e Environm ent for NN M automatically installed ovwd b.auth and ovw .auth files (+ +) by d efault for w eb Discovery via ping sw eep capability through load hosts -c Support for GIF89a im ages on the m aps ovspm d generates pop -up w hen N N M service term inates N ew Contributed Applications: Java-based grapher, Web-based Report Presenter, H igh Availability Support/ Service Guard (H P-UX only) Wind ow s only: Support for WMI via Rem ote Power On DMI 2.0 Support via Intel's DMI 2.0 Service Provid er version 1.10 and DMI Explorer version 1.11 Support for pixmap format icons Several other form erly UNIX-only features available N N M Pre-6.0 release history: 5.02: 5.01: 5.0: 4.1: 4.0: 1/ 6/ 3/ 3/ 8/ 16/ 98 23/ 97 1/ 97 96 95 3.31: 3.1: 3.0: 2.0: 1.0: 11/ 93 7/ 92 5/ 92 2/ 91 6/ 90 Product feature deltas by OS N N M shares a com m on cod e base am ong the fou r su p p orted op erating system s, so OS d ifferences are few , bu t here are som e: OS-specific m enu bar item s, e.g., Registry Ed itor in Window s and Emanate agent m etrics in UN IX, access to SAM in H P-UX, etc. DMI Brow ser (ovcapsd) and RDMI d iscovery (IPX) is only available in Window s version 295 Fognet’s Field Guide to OpenView NNM UN IX versions use Apache Web Server, Wind ow s uses IIS UN IX versions support Oracle, Wind ow s, SQL Server 2000 UN IX supports ovwperms com m and , Wind ow s d oesn‟t Syslog feature in N N M 7.01+ not available on Wind ow s Linux platform only supports NNM Stand ard Ed ition until V7.53 Linux: integration w ith Custom er View s not supported Linux & Wind ow s: H igh availability integrations not supported Linux supports only the OV_NodeIf event correlation Linux does not support ovperm s.ovpl N N M relies heavily on ind ivid ual OS host nam e lookup algorithm s, w hich d iffer and can d irectly im pact performance. UN IX submaps all persistent, Wind ow s: som e are transient IPV6 is only supported on UN IX N N M AE installations OS and softw are support matrix N N M 8.01 2003 Ent x64 SP2 2003 Ent x64 R2 SP2 H P-UX 11iv3 Solaris 10 SPARC Red Hat AS 4.0 Red Hat ES 4.0 Optional External DB‟s: Oracle 10.2.0.x N N M 8.00 2003 Ent x64 SP2 2003 Ent x64 R2 SP2 H P-UX 11iv3 Red Hat AS 4.0 Red Hat ES 4.0 Optional External DB‟s: Oracle 10.2.0.x N N M 7.53 2000 Pro, Srv, Ad v Srv, Term Serv, (SP4), XP Pro SP2, 2003 Std , Ent, SP1 or SP2 2003 R2 Std , Ent H P-UX 11.0, 11.11 (11iv2), 11.23 (11iv2 PA or IT), 11.31 (11iv3 PA or IT) N ote: H P-UX 11iv3 requires OVSN MP Emanate Agent Solaris 2.8 2.9 2.10 Red hat Linux AS 2.1, Upd ate 1 (2.4.9-e.3), Update 2 (2.4.9-e.24) Red H at Enterprise Linux 4, Update 2, Update 5 (64 bit) N ote: Linux variants support NNM Starter Ed ition only Java Plug-in: 11.4.2(Solaris 8), 1.4.2_02(H P-UX 11.0), 1.4.2_04(Red Hat AS2.1), 1.4.2_05(H P-UX 11iv2-PA), H P-UX 11.23-IT), 1.4.2_06(Solaris 10)1.4.2_10(H P-UX 11iv3-PA&IT), 1.5(All except Wnd ow s XP) Web Srvs: Apache, Apache Stronghold , Raven, MS Peer Web Services, MS IIS 4.0, 5.0 Optional External DB‟s: Oracle 8.1.7(H P-UX 11.11, Red Hat AS2.1, Solaris 9, Wind ow s 2003), 9.2.0.2(H P-UX 11.23-IT), 9.2.0.3(H P-UX 11.11, Red Hat AS 2.1 & EL4, Solaris 9, Window s 2003), 9.2.0.6 (Solaris 10), 10gR1(HP-UX 11.0, & 11iv2-PA, Red Hat EL4, Solaris 8 & 10, Window s XP), 10gR2(H PUX 11.11 & 11.23-IT, red Hat EL4, Solaris 9 & 10, Window s 2000), MS SQL 2000-SP3(Window s 2003), MS SQL 2005 (Wind ow s 2003) N N M 7.51 (*** = w ith Intermediate Patch 15; **** = w ith IP 18) 296 Product Details 2000 Pro, Srv, Ad v Srv, Term Serv, (SP4), XP Pro, 2003 Std , Ent H P-UX 11.0, 11.11 (11iv2) Solaris 2.8 2.9 2.10 Red hat Linux Advanced Server 2.1 (SE only) Java Plug-in: UN IX: 1.4.2(Sun/ mozilla), 1.4.2_02(H P-UX), 1.4.2_04(Linux), 1.4.2_05(H P 11.23), 1.4.2_06(Solaris 10), 1.5***, 1.5**** Wind ow s: 1.4.2, 1.5*** Web Srvs: Apache, Apache Stronghold , Raven, MS Peer Web Services, MS IIS 4.0, 5.0 Optional DB‟s: Oracle 8.1.7(not 11.23), 9.2.0.2(N ot Wind ow s), 9.2.0.3(not 11.23), 10gR1, 10gR2, MS SQL 2000 (SP3) N N M 7.5 2000 Pro, Srv, Ad v Srv, Term Serv, (SP4), XP Pro, 2003 Std , Ent H P-UX 11.0, 11.11 (11i), (11.23 Itanium w / SE Only) Solaris 2.8 2.9 Red hat Linux Advanced Server 2.1 (SE only) Java Plug-in: Sun: 1.4.1(netscape), 1.4.2(m ozilla), Wind ow s: 1.4.2, H P: 1.4.2_02, Linux: 1.4.2_04 Web Srvs: Apache, Apache Stronghold , Raven, MS Peer Web Services, MS IIS 4.0, 5.0 Optional DB‟s: Oracle 8.1.7(not 11.23), 9.2.0.2(N ot Wind ow s), 9.2.0.3(not 11.23), MS SQL 2000 (SP3) N N M 7.01 2000 Pro, Srv, Ad v Srv, Term Serv, (SP1-4),XP Pro H P-UX 11.0, 11.11 (11i), (11.23 Itanium w / SE Only) Solaris 2.8 2.9 Red hat Linux Advanced Server 2.1 Java Plug-in: H P-UX: 1.4.1.05, others: 1.4.2_01 Web Srvs: Apache, Apache Stronghold , Raven, MS Peer Web Services, MS IIS 4.0, 5.0 Optional DB‟s: Oracle 8.1.7(not 11.23), 9.2.0.2(N ot Wind ow s), 9.2.0.3(not 11.23), MS SQL 2000 (SP3) N N M 6.41 2000 Pro, Srv, Ad v Srv, Term Serv, (SP1-4*), XP Pro H P-UX 11.0, 11.11 (11i) Solaris 2.8 2.9 Java Plug-in: 1.4 Web Srvs: Apache, Ap ache Stronghold , Raven, MS Peer Web Services, MS IIS 4.0, 5.0 Optional DB‟s: Oracle 8.1.7, 9.2.0.1, MS SQL 2000 (SP2) N N M 6.31 2000 Pro, Srv, Ad v Srv, Term Serv, (SP1-3), N T 4.0 (SP6a) H P-UX 11.0, 11.11 (11i) Solaris 2.6, 2.7, 2.8 Java Plug-in: 1.3.1 or 1.4 Web Srvs: Apache, Apache Stronghold , Raven, MS Peer Web Services, MS IIS 4.0, 5.0 Optional DB‟s: Oracle 8.0.6, 8.1.7, 9.0.1 , MS SQL 7.0 (SP3), 2000 (SP2) 297 Fognet’s Field Guide to OpenView NNM N N M 6.2 2000 Pro, Srv, Ad v Srv, Term Serv, (SP1-4*), N T 4.0 (SP6a) H P-UX 10.2, 11.0, 11.11 (11i) Solaris 2.6, 2.7, 2.8 Java Plug-in: Varies by platform for CDP, Nod e View s only Web Srvs: Apache, Apache Stronghold , Raven, MS Peer Web Services, MS IIS 4.0, 5.0 Optional DB‟s: Oracle 7.3.4(10.20 only), 8.0.6, 8.1.7, MS SQL 7.0 (SP3), 2000 (SP2) N N M 6.1 2000 Srv, Ad v Srv, Term Serv, (SP1), N T 4.0 (SP4,5,6a) H P-UX 10.2, 11.0 Solaris 2.51, 2.6, 2.7, 2.8 (Requires installation utilities from ftp:/ / ovw eb.external.hp.com/ pub/ dow nload/ installs) Web Srvs: Apache, Apache Stronghold , Raven, MS Peer Web Services, MS IIS 2.0, 3.0, 4.0, 5.0 Optional DB‟s: Oracle 7.2.3.0, 7.3.4, 8.0.4, 8.0.5, 8.0.6, 8.1.5, 8.1.6, MS SQL 6.5, 7.0 N N M 6.01 N T 4.0 (SP3,4,5,6a) H P-UX 10.2, 11.0 Solaris 2.51, 2.6, 2.7 (Requires installation utilities from ftp:/ / ovw eb.external.hp.com/ pub/ dow nload/ installs) Web Srvs: Apache, MS Peer Web Svcs, MS IIS 2.0, 3.0, 4.0 Optional DB‟s: Oracle 7.2.3.0, 7.3.4, 8.0.4, 8.0.5, MS SQL 6.5, 7.0 N N M 6.0 N T 4.0 (SP3,4,5,6a) H P-UX 10.2, 11.0 Solaris 2.51, 2.6 (Requires installation utilities from ftp:/ / ovw eb.external.hp.com/ pub/ dow nload/ installs) Web Srvs: Apache, MS Peer Web Svcs, MS IIS 2.0, 3.0, 4.0 Optional DB‟s: Oracle 7.2.3.0, 7.3.4, 8.0.4, MS SQL 6.5, 7.0 * With Patch ** Not on Window s Standard edition vs. Advanced edition AE includ es the Problem Diagnostics prod uct AE Provid es Duplicate IP Support for OADs AE Supports new APA Poller (Active Problem Analyzer) AE includ es Ad vanced Routing SPI for OSPF, H SRP, IPV6 N o Distribution features in SE, i.e., ovrepld, xnmtopoconf, but an SE instance can act as an N N M collection station AE prod uct support matrix for Level 2 topology at: openview.hp.com/ prod ucts/ nnmet/ support/ d evice_support.htm l N ode definition N N M cou nts nod es tow ard s the license lim it by inclu d ing: 298 Product Details IP ad d ressable nod es w ith SN MP agents IP ad d ressable nod es w ithout SNMP agents Level 2 nod es d iscovered by netmon IPX nod es on Wind ow s platform s N odes in OAD‟s d iscovered by ET (AE only) Any nod e that is u nm anaged (beige) is not cou nted tow ard the license nod e cou nt. To obtain a nod e cou nts, ru n: ovotopodump –l (ell) A chassis w ith m u ltip le SN MP agents m ay cou nt as m u ltip le nod es. N on-SN MP entities that share a DN S nam e are consid ered a single nod e. Sim ilarly, if interfaces on a non -SN MP read able d evice have d istinct DN S nam es, and N N M can‟t otherw ise d eterm ine w ith w hat nod e they are associated , they m ay be treated as sep arate nod es. This can u su ally be straightened by ad d ing DN S A and / or CN AME record s for the interfaces. Ad d itional am bigu ities m ay occu r on nod es that are assigned m u ltip le ad d resses for a single interface or w here IP ad d resses float betw een interfaces. It is recom m end ed to p lace su ch ad d resses in the netm on.noDiscover file and d elete them from the top ology. Inconsistent resp onses to SN MP m ay occu r w hen Rou nd Robin DN S is being u sed to m ap requ ests to m u ltip le ap p lication servers. If m anaging IPV6, the IPV6 ad d ress cou nts tow ard s the license cou nt for that p lu g-in, and it consu m es an ad d itional nod e cou nt for the associated IPV4 ad d ress, w hich is requ ired for each IPV6 ad d ress to be m anaged . The follow ing ad d itional com m and s m ay p rod u ce nod e cou nts that conflict w ith the resu lts from the ovotopodump com m and above, bu t these com m and s are not u sed by the licensing system in d etem ining nod e cou nts. ovet_topodump.ovpl –info snmpget <NNM-server> nodeCount.0 ovstatus –v netmon (polled interfaces) ovobjprint -S (Objects) To p revent nod es from being d iscovered tow ard the nod e cou nt, see the section on lim iting d iscovery on p age 39. To d iscover nod es, then u nm anage then, so m ore nod es w ill show u p on the m ap s bu t not cou nt tow ard s the nod e cou nt, see the section on u nm anaing objects on p age 41. 299 Fognet’s Field Guide to OpenView NNM N N M smart plug-ins N N M SPI for Ad vanced Routing H SRP, OSPF, and IPV6 Management. Requires AE and separate license, 60-day d em o installed by d efault w ith AE N N M SPI for IP Telephony (One for Avaya and one for Cisco) N N M SPI for LAN/ WAN Ed ge; form erly Fram e Relay SPI N N M SPI for MPLS VPN N N M SPI for IP Multicast HP Softw are product suite summary This list is lim ited to only the m ost p op u lar IT m anagem ent ap p lications w ithin the Op enView su ite, and it d oes not inclu d e H P‟s offerings in the storage m anagem ent, m essaging or telecom m anagem ent sp aces. After acqu iring Peregrine and Mercu ry in 2006 and 2007, H P rebrand ed all its softw are to rem ove the Op enView nam e. The below listing d oes not inclu d e all the new offerings from those verd ors, bu t the Peregrine Service Center p rod u ct is now heavily integrated w ith H P Service Desk offering. Service Desk (OVSD) H elp d esk application, ITSM/ ITIL com pliant. H P acquired Prolin in 1997 N etw ork Configu ration Manager (N CM) Multivend or configuration and com pliance m anager. Keeps track of configurations and changes for netw ork d evices. Checks if configurations are com pliant w ith com pany stand ard s. OEM-ed from Voyence Internet Services (OVIS) SLA-oriented monitoring via softw are probes for w eb services Transaction Analyzer (OVTA) Monitoring and d iagnostics for both w eb and non -web J2SE, J2EE, .NET, and COM+ Op erations (OVO) A d istributed system s and application managem ent tool, supporting a num ber of Sm art Plug-Ins for application m anagement (SPIs). Operations for UN IX (OVOU) has an embed d ed copy of NN M. Operations for Wind ow s (OVOW) does not includ e NN M. Form erly Vantage Point Operations (VPO). Form erly IT/ Operations (ITO). Form erly Operations Center (OPC). Form erly ManageX (acquired ManageOne 1999) Service N avigator Built into OVOW, and bund led w ith OVOU to provid e service-oriented hierarchical view s. No add itional license required . Service Inform ation Portal (SIP) Web-based portal for serving secure view s of d ata from many OpenView prod ucts includ ing OVOU, OVOW, N N M, OVIS, Service N avigator, OVPI, OVPM, Reporter, OVSD and Data Protector. Perform ance Insight (OVPI) 300 Product Details N etwork perform ance reporting tool. Form erly Trend ; H P acquired Trinagy in 2001 Cu stom er View s Requires NN M; logically organizes netw ork elem ents Canned reports for Frame, ATM, SON ET, DS-1/ E1, DS-3/ E3 Perform ance Su ite GlancePlus: Realtim e stats for H P-UX, Solaris, AIX, & Linux Agent: Gathers and trend s system stats, form erly MeasureWare. Manager: Gather, d isplay d ata from agents, form erly PerfView . Rep orter Extensible reporting application generates pre-d efined Web reports from Perform ance agents and OVO. Problem Diagnostics (PD) Software probe-based network path analysis Bund led into NN M 7.0+ AE; no longer a separate prod uct Event Correlation Services (ECS) State m achine for event correlation runtim e bund led w ith NN M since 6.0; Correlation Com poser ad d ed in 6.41 allow s d evelopm ent of ECS logic; ECS Designer is a separate prod uct for Full ECS circuit d evelopm ent capabilities Rou te Analytics Managem ent System (RAMS) IP Routing analysis and trend ing probes; Separate Prod uct, works only w ith N N M 7.5 AE; OEM‟s from Packet Design in 2004. SN MP Research‟s SN MP Secu rity Pack. Separate, non-H P add -on for SN MPv3 support. Bund led w ith NN M 7.5+ (AE only). Extensible SN MP Agent Standalone extensible agent for HP-UX or Solaris only. Rad ia H P acquired Novad igm in 2004 to fill out their gap in change, configuration and release m anagem ent. Data Protector. Enterprise backup application. Form erly Om niBack Oracle for Op enView H P-bund led Oracle database for supported OpenView apps. PolicyXp ert Provid es packet-level band w id th m etering and QOS man agem ent. Insight manager integration System s Insight, form erly Com p aq Insight Manager (CIM) Integration: http:/ / h18013.ww w1.hp.com / prod ucts/ servers/ managem ent/ openview / ind ex.htm l All SIM d ow nload s: http:/ / h18000.ww w1.hp.com / prod ucts/ servers/ managem ent/ hpsim / do w nload .htm l 301 Fognet’s Field Guide to OpenView NNM Core N N M product numbers (U.S), versions 6.4-7.51 Product N umber All operating systems: Version D escription J5323BA J5324BA T2489AA/ BA T2490AA/ BA T2578AA/ BA 6.4 6.4 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x Manuals, English Med ia Med ia Starter Ed ition Manuals, English Ad vanced Ed ition Manuals, Eng. J5316BA J5315BA J5330BA J5328BA J5329BA J1249AB J1253UB J5310BA T2484AA/ T2646BA T2688BA T2491AA/ T2692BA T2700AA/ T2704AA/ T2708AA/ T2712AA/ T2637AA/ T2640AA/ T2643AA/ T2651BA T2495AA/ T2500AA/ T2505AA/ T2511AA/ T2579AA/ 6.4 6.4 6.4 6.4 6.4 6.4 6.4 6.4 7.0/ 7.5 7.5 7.0/ 7.5/ 7.0/ 7.0/ 7.0/ 7.0/ 7.0/ 7.0/ 7.0/ 7.5/ 7.0/ 7.0/ 7.0/ 7.0/ 7.0/ N N M 250 Ed ition Enterprise Ed ition (Unlim ited ) Upgrad e N N M 250 to 6.4 250 Upgrad e N N M 250 to 6.4 Entp Upgrad e N N M Entp to 6.4 Entp N N M 250 Nod e Increm ent N N M 250 Nod e Incr. Upgrad e Developer‟s Toolkit Starter Ed ition 250 Nod es (SE) Starter Ed ition Unlim ited Upgrad e SE 250 to SE Unlim ited Upgrad e SE 250 to AE 250 Upgrad e SE Unltd to AE 1000 Upgrad e AE 250 to AE 1000 Upgrad e AE 1000 to AE 5000 Upgrad e AE 1000 to Unlim ited Upgrad e AE 5000 to Unlim ited Upgrad e 6.x 250 to 7.0/ 5 SE 250 Upgrad e 6.x 250 to 7.0/ 5 AE 250 Upgrd 6.x Enpr to 7.0/ 5AE 1000 Upgrd 6.x Enpr to 7.0/ 5SE Unltd Ad vanced Ed ition 250 N od es Ad vanced Ed ition 1000 Nod es Ad vanced Ed ition 5000 Nod es Ad vanced Ed ition Unlim ited Developers Toolkit HP-UX: BA BA BA BA BA BA BA BA BA BA BA BA BA BA 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x 7.5x Solaris: J5320BA J5317BA J5327BA J5397BA J5325BA J1256AB J1260UB J5311BA T2485AA/ BA T2647BA T2689BA T2492AA/ BA 302 6.4 6.4 6.4 6.4 6.4 6.4 6.4 6.4 7.0/ 7.5 7.5 7.5 7.0/ 7.5 N N M 250 Ed ition Enterprise Ed ition (Unlim ited ) Upgrad e N N M 250 to 6.4 250 Upgrad e N N M 250 to 6.4 Entp Upgrad e N N M Entp to 6.4 Entp N N M 250 Nod e Increm ent N N M 250 Nod e Incr. Upgrad e Developer‟s Toolkit Starter Ed ition 250 Nod es (SE) Starter Ed ition Unlim ited Upgrad e SE 250 to SE Unlim ited Upgrad e SE 250 to AE 250 Product Details T2693BA T2701AA/ T2705AA/ T2709AA/ T2713AA/ T2638AA/ T2641AA/ T2644AA/ T2652BA T2496AA/ T2501AA/ T2506AA/ T2512AA/ T2580AA/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.5/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x Upgrad e SE Unltd to AE 1000 Upgrad e AE 250 to AE 1000 Upgrad e AE 1000 to AE 5000 Upgrad e AE 1000 to Unlim ited Upgrad e AE 5000 to Unlim ited Upgrad e 6.x 250 to 7.0/ 5 SE 250 Upgrad e 6.x 250 to 7.0/ 5 AE 250 Upgrd 6.x Enpr to 7.0/ 5AE 1000 Upgrd 6.x Enpr to 7.0/ 5SE Unltd Ad vanced Ed ition 250 N od es Ad vanced Ed ition 1000 Nod es Ad vanced Ed ition 5000 Nod es Ad vanced Ed ition Unlim ited Developers Toolkit BA BA BA BA BA 6.4 6.4 6.4 6.4 6.4 6.4 6.4 6.4 7.0/ 7.5x 7.5x 7.5x 7.0/ 7.5x 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x 7.0/ 7.5x N N M 250 Ed ition Enterprise Ed ition (Unlim ited ) Upgrad e N N M 250 to 6.4 250 Upgrad e N N M 250 to 6.4 Entp Upgrad e N N M Entp to 6.4 Entp N N M 250 Nod e Increm ent N N M 250 Nod e Incr. Upgrad e Developer‟s Toolkit Starter Ed ition 250 Nod es (SE) Starter Ed ition Unlim ited Upgrad e SE 250 to SE Unlim ited Upgrad e SE 250 to AE 250 Upgrad e SE Unltd to AE 1000 Upgrad e AE 250 to AE 1000 Upgrad e AE 1000 to AE 5000 Upgrad e AE 1000 to Unlim ited Upgrad e AE 5000 to Unlim ited Upgrad e 6.x 250 to 7.0/ 5 SE 250 Upgrad e 6.x 250 to 7.0/ 5 AE 250 Upgrd 6.x Enpr to 7.0/ 5AE 1000 Upgrd 6.x Enpr to 7.0/ 5SE Unltd Ad vanced Ed ition 250 N od es Ad vanced Ed ition 1000 Nod es Ad vanced Ed ition 5000 Nod es Ad vanced Ed ition Unlim ited Developers Toolkit BA BA BA BA 7.5x 7.5x 7.5x 7.5x Starter Ed ition 250 Nod es Starter Ed ition Unlim ited Upgrad e SE 250 to SE Unlim ited Developers Toolkit BA BA BA BA BA BA BA BA BA BA BA BA Window s: J5321BA J5321BA J5314BA J5398BA J5326BA J1242AB J1242UB J5312BA T2486AA/ T2648BA T2690BA T2493AA/ T2694BA T2702AA/ T2706AA/ T2710AA/ T2714AA/ T2639AA/ T2642AA/ T2645AA/ T2653BA T2497AA/ T2502AA/ T2507AA/ T2513AA/ T2581AA/ BA BA BA BA BA BA BA BA BA Linux: T2487AA/ T2649AA/ T2691AA/ T2582AA/ N otes: Prod u ct nu m bers end ing in AA refer to 7.0/ 7.01 versions and those end ing in BA refer to version 7.51 and 7.53. Prod u ct nu m bers for N N M 6.4 and 6.41 are the sam e and p rod u ct nu m bers for N N M 7.0 and N N M 7.01 are the sam e. 303 Fognet’s Field Guide to OpenView NNM Support package numbers (N orth America only) A new su p p ort stru ctu re w as introd u ced at the end of 2003. Several com p anies d eliver SPI-su p p ort on behalf of H P, and have ad d -ons to the su p p ort p ackages from H P. Those are not listed here. Old Support Options OS2, OS3 3Y2, 3Y3 OS6 3Y6 OS3 3Y3 OS6 3Y6 n/ a n/ a OTx n/ a U5008AA 005 012 025 304 N ew H P Care Pack #s H A106A1 H A106A3 H A107A1 H A107A3 H A109A1 H A109A3 H A110A1 H A110A3 H A111A1 H A111A3 H A112A1 H A112A3 H A287A1 n/ a n/ a n/ a Description Software Sppt – Business hrs, upd ates, 1 yr Software Sppt - Business hrs, upd ates, 3 yrs Software Sppt – 24x7, updates, 1 yr Software Sppt – 24x7, updates, 3 yrs SW & H W sppt - Business hrs, upd ates, 1 yr SW & H W sppt - Business hrs, upd ates, 3yrs SW & H W sppt – 24x7, upd ates, 1 yr SW & H W sppt - 24x7, updates, 3yrs Old PSS (Personalized system sppt) – 1 yr Old PSS (Personalized system sppt) – 3 yrs Old CSS (Critical System s sppt) – 1 yr Old CSS (Critical System s sppt) – 3 yrs OpenView Prem ier Services U2461AA Developer Support 5-pack U2461AA Developer Support 12-pack U2461AA Developer Support 25-pack Glossary A ACL AE AIX Alarm Alp ha AMI AN SI APA ARF ARP ASN .1 ATM B8ZS BECN BER BES BGP BIN D BRI BSD CA CERT CCO CDC CDP CFA CGI CISC CIDR CIM CIR CLI CN AME DN S A Record or Ad d ress Record p er RFC 1035 Access Control List, sets access rights (MS, Cisco, etc) Acronym for H P Op enView N N M Ad vanced Ed ition IBM‟s p rop rietary UN IX system V variant An N N M d ep iction of an event conveying relations, etc. A 64-bit chip architectu re introd u ced by DEC in 1992 Alternate Mark Inversion, an old er DDS T1 encod ing Am erican N ational Stand ard s Institu te, w w w .ansi.org Active Problem Analyzer, N N M‟s ET-based Poller N N M Ap p lication Registration Files, p age 6 Ad d ress Resolu tion Protocol p er RFCs 826, 1293 Abstract Syntax N otation One, an OSI-d efined syntax Asynchronou s Transfer Mod e p er RFC 1932 Binary 8 Zero Su bstitu tion, Clear Channel T1 cod ing A FR Backw ard Exp licit Congestion N otification Basic Encod ing Ru les are ASN .1 encod ing ru les Binary Event Store, N N M‟s raw event store Bord er Gatew ay Protocol, an EGP p er RFC 1771 Berkeley Internet N am e Dom ain, DN S, w w w .isc.org Basic Rate Interface, a typ e of ISDN interface Berkeley Softw are Distribu tion, w w w .bsd .org Com p u ter Associates, Inc. w w w .ca.com Com p u ter Em ergency Resp onse Team , w w w .cert.org Cisco Connection Online – Cisco registered su p p ort site Cisco Discovery Configu ration, controls ET d iscovery p aram s Cisco Discovery Protocol, ad vertises Cisco d evice d ata N N M‟s Connectivity Fau lt Analyzer, an APA su bsystem Com m on Gatew ay Interface for Web, RFC 3875 Com p lex Instru ction Set Com p u ting CPU architectu re Classless Inter-Dom ain Rou ting, RFC 1518, 1519, 1817 Com p aq Insight Manager, now called SIM Com m itted Inform ation Rate, see Fram e Relay Com m and Line Interface, accesses OS com m and s DN S Canonical N am e Record (Alias), RFC 1035 305 Fognet’s Field Guide to OpenView NNM CS CSOV CSV CW2K CWSI DB DBM DBMS DDS DE DEC DFM DIM DOD DOS DMI DMTF DMZ DN S DP DS1, DS3 Du p lex DW E1/ E3 ECS EGP Envvars Event ET Ethernet FDB FECN FQDN Fram e FR grep GN U GPL GUI HC HP H P-UX H POV 306 N N M Collection Station: a DIM term Chip Su tton‟s OV Perl API Mod u le, w w w .cs-net.com Com m a Sep arated Variable text file for DB im p orts CiscoWorks 2000, Cisco‟s rou ter m anagem ent tool CiscoWorks for Sw itched Internetw orks; p art of CW2K Generic term for Data Base UCB‟s flat file d atabase libraries; em bed d ed in UN IX Database Managem ent System , e.g. Oracle, SQL Server Datap hone Digital Service, AT&T‟s p re-T1 link service Discard Eligible Fram e Relay p acket Digital Equ ip m ent Corp . A d efu nct com p u ter com p any CiscoWork‟s Device Fau lt Manger, su bset of SMARTS N N M Distribu ted Internet Discovery United States Dep artm ent of Defense Denial of Service attack or MS Disk Op erating System Desktop Managem ent Interface, a DMTF stand ard Desktop Managem ent Task Force, a stand ard s bod y Dem ilitarized Zone betw een internet and intranet Dom ain N am ing Service; Many RFCs, p rim arily 1035 H P Op enView Data Protector; enterp rise backu p tool 1.544Mbit(t1), 43Mbit(t3) serial lines, RFC 1232, 2495 Synonym ou s w ith Fu ll Du p lex; bi-d irectional traffic Op enView N N M d ata w arehou se, a relational DB See DS1/ DS3 Op enView Event Correlation Services p rod u ct Exterior Rou ting Protocol, generic for internet rou ting Contraction of environm ent variables N N M-encap su lated trap con veying statu s, sou rce, etc Extend ed Top ology, N N M‟s Level 2 top ology tool A LAN architectu re d efined by IEEE 802.3 stand ard Acronym for Brid ge Fow ard ing Database, A FR Forw ard Exp licit Congestion N otification Acronym for Fu lly Qu alified Dom ain N am e The OSI Layer 2 (d ata link) encap su lation Fram e Relay, WAN p acket sw itching, RFC 1315, 2115 UN IX com m and to search for a p attern Short for GN U‟s N ot UN IX, a free OS, w w w .fsf.org Short for GN U Pu blic License, w w w .gnu .org also UI, an acronym for Grap hical User Interface H igh Cap acity MIB variables su p p ort 64-bits cou nters H ew lett-Packard Corp ., a large com p u ter com p any H P‟s p rop rietary UN IX Sytem V variant H ew lett-Packard Op enView , synonym ou s w ith OV Glossary H SRP IAN A IBM ICMP IGP IEEE IE If IfMIB IIS IOS IETF IP IPV4 IPV6 ISO ISDN IT IT ITRC J2EE Java JPI JRE JVM Kerm it LAN LRF MAC Man MAU MD5 MIB m ib2 MibExp r MLT Motif MS MTU N AT N DBM N DE N etbios N etBT H ot-Sw ap p able Rou ting Protocol, A Cisco p rotocol Internet Assigned N u m bers Au thority, w w w .iana.org Int‟l Bu siness Machines, a large com p u ter com p any Internet Control Message Protocol, Packet IP, RFC 792 Interior Gatew ay Protocol, generic for intranet rou ting Int‟l Institu te of Electronic Engineers, w w w .ieee.org Microsoft‟s Internet Exp lorer w eb brow ser Contraction for Interface, see N IC Interface Mib, RFC 1573, 2233 Microsoft Internet Inform ation Server, a w eb server Internal Op erating System ; Cisco et al. Internet Engineering Task Force, w w w .ietf.org, see RFC Internet Protocol, See RFCs 791, 793, 1180, etc Internet Protocol Version 4, 32-bit IP ad d r, RFC 791 Internet Protocol Version 6, 128-bit IP ad d r, RFC 2460 International Org. for Stand ard ization, w w w .iso.org Integrated Services Disgital N etw rok; a telep hony netw ork Acronym for Inform ation Technology Short for Itaniu m in OS softw are su p p ort m atrix (p age 296) H P‟s IT Resou rce Center; online d b at w w w .itrc.hp .com Java 2 Platform Enterp rise Ed ition, d ev p latform from Su n Su n‟s high-level object-oriented p rogram m ing langu age Java Plu g-In, equ ivalent to JRE Java Ru ntim e Environm ent, http :/ / java.su n.com / j2se/ Java Virtu al Machine, a virtu alized JAVA p rocessor Serial com m u nications from w w w .colu m bia.ed u / kerm it Acronym for Local Area N etw ork N N M Local Registration File Med ia Access Control Ad d ress, AN SI/ IEEE Std 802 UN IX Man p ages p rovid e online help facilities Med ia Access Unit, e.g an Ethernet transceiver Message Digest Encryp tion algorithm , RFC 1321 SN MP Managem ent Inform ation Base, p er RFC 1212 MIB hold ing the m ost com m on agent d ata, RFC 1213 N N M Mib Exp ression, com bines variables u sing RPN Mu lti-Link tru nking, an aggregated p ort p rotocol for N ortel OSF GUI gu id elines for look and feel, op engrou p .org N N M Managem ent Station (DIM); also: Microsoft Maxim u m Transm ission Unit. A p acket size lim itation N etw ork Ad d ress Translation, RFC 1631 et al. A flat file DB library set u sed by N N M based on DBM Cisco‟s N etflow Data Exp ort, a sw itch traffic m onitor N etw ork Basic Inp u t/ Ou tp u t System , A Microsoft API N etbios over TCP/ IP, an old er Wind ow s service 307 Fognet’s Field Guide to OpenView NNM N etm ask netmon N etView N IAP N IC N IS N od e N MS NNM N ortel OAD Object Octet ODBC OEM OID OOID Op enSSH Op enView OQL OUSPG OV OVO OVOW OVPI OVR ovw OS OSF OSF/ 1 OSI OSI OSPF Packet PA PAgP Perl PID p ing PD PDU PPP PPTP 308 Contraction of IP Su bnet netw ork m ask, RFC 950 N N M‟s d iscovery and p olling d aem on An IBM N MS based in p art on Op enView cod e A US Govt IT secu rityt initiative, see niap -ccevs.org N etw ork Interface Card , a m ed ia access d evice N etw ork Inform ation Service (Yellow p ages) (Su n) A system or d evice connected to a netw ork A generic term for N etw ork Managem ent System H ew lett-Packard ‟s Op enView N etw ork N od e Manager A large p rovid er of netw ork equ ip m ent Overlap p ing Ad d ress Dom ain (TCP/ IP) In N N M, a u niqu ely id entified object d atabase entity A single byte of d ata, or 8 bits Op en Data Base Connectivity; SQL Access Grou p Std Original Equ ip m ent Manu factu rer‟s em bed d ed p rod u ct Object ID, d otted nu m ber for SN MP MIB variable Op enView Object ID, u niqu e no. assigned in object d b Secu re op en sou rce logon tool from w w w .op enssh.com H P‟s su ite of enterp rise m anagem ent softw are p rod u cts A su bset of SQL w ith H P extensions for N N M/ ET Ou lu Univ. Secu re Program m ing Grou p , ee.ou lu .fi/ ?en A contraction of Op enView , a su ite of H P softw are Op enView Op erations for UN IX (A system s m anager) Op enView Op erations for Wind ow s Op enView Perform ance Insight (form erly Trend Micro) Op enView Rep orter Prod u ct OV Wind ow s, the N N M static Map foregrou nd p rocess Op erating System , e.g. Solaris, Wind ow s 2000, H P-UX Op en Softw are Fou nd ation, now w w w .op engrou p .org Prop rietary 64-bit UN IX variant from DEC; now Tru 64 Op en System Interconnect, set of ISO-d efined p rotocols Ontario Sw ine Im p rovem ent, Inc – w w w .osi.org Op en Shortest Path First IGP rou ting, RFC 1583,2328 See PDU Short for PA-RISC, H P‟s CPU chip architectu re Cisco‟s Port Aggregation Protocol Practical Extraction and Rep orting (scrip ting) Langu age Process ID. A u niqu e nu m ber assigned by the OS A p rogram that issu es ICMP Echo requ ests and rep lies H P Op enView Problem Diagnosis, an N N M bu ilt -in Package Data Unit, the OSI Layer 3 d ata encap su lation Point-to-Point Protocol, RFC 1661, 1990 (Mu ltilink) etc. Point-to-Point Tu nneling Protocol, RFC 2637 Glossary PRI PTP PTR RARP RAMS RAN CID rc Ref RDMI RFC RISC RIP RME RMON RMON II RPN SAA SAM SAN S SE SEA sed SID SIM Sim p lex SMI SMLT Solid SOX SPAN SPI SQL SMTP SN MP SN N P SSH SSH 1,2 SSL syslog T1/ T3 TAC TAP TCP tail Prim ary Rate Interface, a typ e of ISDN interface Acronym for Point To Point DN S Pointer Record for reverse looku p , RFC 1035 Reverse Ad d ress Resolu tion Protocol, RFC 903, 1931 Op enView Rou te Analytics Managem ent System Really Aw esom e N ew Cisco confIg Differ UN IX Rem ote Control files, au to d aem on start and stop Reference p ages. Accessible from N N M H elp m enu H P‟s Rem ote DMI access p rotocol IETF Requ est for Com m ent. Search w eb for RFC 3160 Red u ced Instru ction Set Com p u ting CPU architectu re Rou ting Inform ation Protocol, an IGP p er RFC 1058 Cisco Resou rce Manager Essentials, p art of CW2K Rem ote Monitoring SN MP extensions, RFC 1757 Extend s RMON from layer 2 to 3 and m ore, RFC 2021 Reverse Polish N otation, m ath via Postfix N otation Cisco Service Assu rance Agent; m onitors p erform ance H P-UX‟s System Ad m instration Manager GUI SysAd m , Au d it, N etw ork, Secu rity Inst. w w w .sans.org Refers to H P Op enView N N M Starter Ed ition Solstice Enterp rise Agent, an SN MP agent, su n.com Unix stream ed itor com m and Oracle System Id entifier, u sed to u niqu ely ID a DB Acroynym for H P System s Insight Manager synonym ou s w ith H alf Du p lex; one-w ay transm ission Stru ctu re of Managem ent Inform ation, see RFC 1155 Sp lit Mu lti-Link Tru nking, an aggregate p rotocol for N ortel Em bed d ed DBMS (w ith N N M). w w w .solid tech.com Boston Red Sox, 2004 World Series Cham p ions, Yeeha! Cisco‟s term for p ort sp anning or p ort m irroring Sm art Plu g-In: ad d -on fu nctionality for N N M, OVO Stru ctu red Qu ery Langu age, a DBMS access m ethod Sim p le Mail Transp ort Protocol, RFC 821, 2821, et al Sim p le N etw ork Mgm t Protocol, RFC 1157, 1905 Sim p le N etw ork Paging Protocol, RFC 1568, 1645 Secu re Shell, generic for SSH 1, SSH 2, Op enSSH , etc Secu re UN IX logon tool from SCS, w w w .ssh.com Secu re Sockets Layer, an encryp tion p rotocol, RFC 2246, et al BSD-based error logging facility, RFC 3164 See DS1/ DS3 Cisco Technical Assistance Center (Su p p ort center) IXO/ Telocator Alp hanu m eric Protocol, see SN N P Transm ission Control Protocol, connection -oriented IP A UN IX com m and that d elivers the last p art of a file 309 Fognet’s Field Guide to OpenView NNM telnet Tivoli Tom cat Trap UDP UCB UN IX VAX Varbind VLAN VLSM VN C VOIP VPDN VRRP WMI 310 A term inal em u lation p rogram / p rotocol, RFC 854 et.al. IBM‟s enterp rise m anagem ent p rod u ct su ite Ap ache‟s Jakarta JAVA servlet An SN MP u nsolicited notification, p er RFC 1215 User Datagram Protocol, connectionless IP, RFC 768 University of California, Berkeley A trad em ark of the Op en Grou p , see w w w .u nix.org A p rop rietary CISC-based OS p rod u ced by DEC Contraction of Variable Bind ing: SN MP trap attribu tes Virtu al LAN , logical grou p s of d evices, RFC 3069 Variable Length Su bnet Mask, Internal CIDR,RFC1817 Virtu al N etw ork Com p u ting, w w w .vnc.com Voice Over IP, Many RFC‟s, e.g. 1889, 2543, 2885, 3261 Virtu al Private Dialu p N etw ork, Cisco, RFC 2764, et al Virtu al Rou ter Red u nd ancy Protocol, RFC 2338, 3867, 2787 Intel‟s Wired for Managem ent Initiative Index $OV_FIELDS, 8 $OV_LOG, 13 $OV_LRF, 5 $OV_REGISTRATION, 6 $OV_SNMP_MIBS, 114 .1.3.6.1.4.1.11.2.17.2.2.0, 109 /etc/ncsd.conf, 36 /etc/netmasks, 118 /etc/networks, 211 1IPathServlet.xml, 253 1NeighborServlet.xml, 251 31 and 32 bit networks, 49 32-bit counters, 159 64-bit counter, 159 64-bit counters, 159 Access Path view, 176 Acknowledged alarm, 110 Action Callback, 164 Action Callback metacharacters, 166 Action Callback Special characters, 166 action callback usage considerations, 165 action callbacks - Java Apps, 222 Action callbacks for a subset of objects, 167 Action callbacks on Windows, 168 add customized symbols, 9 Add Objects Manually, 204 Administrative status, 206 Administrative Status Colors, 110 Advanced Edition, 307 Agent in Distress, 115 aggregate port, 81 Aggregate port scenario, 80 Aggregated port support, 257 alarm, 98 Alarm browser, 110 Alarmpoint, 172 ALLOW_ALL file, 169 Allowable characters in a label, 29 allowOverlap, 276 Amerigo, 183 APA - enable, 74 APA - Filtering by ifType, 90 APA - Filters, 86 APA - firewalls, 91 APA - interface configuration changes, 232 APA - interface table additions, 233 apa - paConfig.xml, 83 APA - polling adjustments, 90 APA - Switching routers, 91 APA - topology events, 92 APA - Troubleshooting, 97 APA aggregated port support, 80 APA Architecture, 71, 72, 74 APA board status support, 81, 82 APA class specifications, 85 APA Connected vs unconnected interface status, 94 APA Demand Polling, 75 APA important node filters, 93 APA Interface ICMP polling, 84 APA memory footprint, 94 APA performance, 93 APA pickManagementAddesss, 95 APA Poller, 71 APA polling behavior, 95 APA status events, 76, 77 APA Status events, 79 APA status in ovw, 79 app-defaults, 9 Application defaults update procedure, 9 Application Registration Files, 6 Arpanet, 211 ARP-cache, 116 ARPChgNewPhysAddr, 116 ASN.1, 144 audio, 171 Authentication Failure, 112 Authentication Failure events, 119 311 Fognet’s Field Guide to OpenView NNM AuthFail trap, 120 autozone.ovpl, 226 beep95_lx.sh, 174 BES – increase size, 273 Big Brother, 183 Binary Event Store, 262 BLAT, 172 Bmail, 173 Brass agent, 139 brassd, 142 bridge.noDiscover, 236 Browser cache, 259 cachedb, 262 CAM, 187 Campus Manager, 186 Cannot allocate 128 colors, 21 Cannot convert string, 22 capability fields, 199 Catalyst 2950, 191 CDC, 188 CERT Advisory CA-2002-03, 144 Changing server hostname, 17 Chassis_Temperature, 133 checkDNS.ovpl, 37 checkPollCfg, 78, 95, 238 Chip Sutton, 182 CIA Agent, 152 CIDR, 117 CIM, 162 Cisco, 185 Cisco web links, 196 Cisco chassis event correlation, 133 Cisco discovery configuration, 188 Cisco IOS, 189 cisco IOS commands, 191 Cisco Link Down Trap, 192 Cisco logging levels, 149 Cisco MIBS, 185 Cisco ping MIB, 191 CISCO PIX routers, 115 Cisco Router General Performance Reports, 270 cisco snmp-server view, 190 Cisco syslog messages, 196 Cisco system stats, 194 Cisco Temperature Probes, 195 Cisco VLANs, 187 Cisco's locIf*BitsSec, 193 312 Cisco-specific event corellation circuits, 196 CiscoSwitchSnmp.user.cfg, 188 CiscoView, 186 CiscoWorks, 186 CiscoWorks web Links, 197 classfulness, 116 Classless Inter-Domain Routing, 118 classless routing, 27 classpath, 222 Cluster virtual IP addresses, 289 clusters, 49 Cold Start, 112 Community String Discovery, 143 Compaq Insight, 162 Compaq Insight Manager, 310 Composer: Enhance, 125; Multi-Source, 125 Composer tracing, 138 Compound Status, 206 Concord eHealth, 184 Concord’s SystemEdge, 152 Connection Editor, 234 connectionEdits, 234 Connector down event correlation, 127 Connector_IntermittentStatus, 132 Container View operations, 246 Container Views, 244 Container Views access control, 247 containers.xml, 245 convertBitmaps.ovpl, 221 cooperative-independent DIM Model, 285 Correlation Composer, 124 Correlation Manager, 124 cpio, 285 Cricket, 184 Critical Path Analysis, 61 cron, 262 Crystal Reports, 266 CS-OV PERL, 182 cu, 173 Customer Views, 308, 309 CW2K, 186 CWSI, 186 Data Collection and Thresholds, 153 Data Collection best practices, 154 Data Collection on High Speed links, 159 data collections - ifSpeed issues, 193 Data Collections on VLANs, 160 Index Data Warehouse, 262 Database maintenance, 271 Database Queries, 263, 265 Database Size limitations, 273 Databases, 262 Datagram, 161 debug on cisco routers, 180 De-Dup, 131 De-dup correlation, 130 dedup.conf, 127, 132 DeDuplication correlator, 127 Delete From All Submaps, 202 Delete object, 203 Deleted alarm, 110 deleteOVDB.ovpl, 12 deleteSecondary, 276 DHCP, 37 DHCP filters, 277 DIM, 275 DIM Limitations, 275 DIM Overlap Modes, 276 DIM Set-up in a nutshell, 279 Disable Default Reports, 272 Disable Solid, 272 Disable the Data Warehouse, 268, 269 disableNSVandISV.ovpl, 250 DiscoSnmpHelperSchema.cfg, 228 Discovery, 39 Discovery filters, 277 discovery polling algorithm, 47 Discovery polls, 41 disk space, 148 Distributed Internet Discovery, 275 Distributed NNM, 275 DMI, 151 DMI MIF, 151 DMZ, 293 DNS, 25, 30 DNS - Windows, 35 DNS - A record, 36 DNS – Cache Only, 30 DNS - ISC BIND, 32 DNS – Split Horizon, 31 DNS tracing, 37 DNS-BIND, 31 DnsNbtLookupOrder, 34 DnsPriority, 33 doNotDiscoverDuplicates, 50 dot1dBaseNumPorts, 187 dot1dTpFdbTable, 187 DS3, 159 Dupip, 254 dupip.conf, 254 duplicate ifAlias, 28 Duplicate ifAlias, 119 Duplicate_IP_Address, 119 Duplicate_IP_Address events, 118 DuplicateIfAlias, 119 dvclient.jnlp, 243 dvUsersManager.ovpl, 243 dynamic views, 198 Dynamic views, 240 Dynamic Views, 240 Dynamic Views - Troubleshooting, 259 Dynamic views access control, 243, 244 Dynamic Views configuration, 242 Dynamic Views edge connectivity, 260 Dynamic Views features, 240 Dynamic Views registration files, 260 Dynamic Views URLs, 241, 242 dynamic views-based alarm browser, 261 dynamicViews.conf, 242 dynamicViewsUsers.xml, 234, 243 ECS - Permanently Disable, 135 ECS Designer, 126 ECS engine logging, 136 ECS logging, 136 ECS Manager commands, 135 ECS runtime, 123 ECS runtime engine, 126 ECS Tracing, 137 ecscmg.html, 124 ecsevgen, 137 ecsevout, 137 ecsmgr, 127, 135 Edge enPortal and nVision, 183 EGP Neighbor Loss, 112 elm, 170 Email notifications, 169 Email notifications and DNS, 171 Emanate, 152 Emanate extension agent, 141 enableNSVandISV.ovpl, 250 enterprise default, 114 entLastChangeTime, 232 Environment Variables, 3, 4 epoch time, 163 ET, 225 313 Fognet’s Field Guide to OpenView NNM ET - connections for unsupported devices, 236 ET - End nodes, 235 ET – importing non-SNMP devices, 236 ET and DNS issues, 239 ET and ovw database synchronization, 231 ET autozoning, 226 ET command summary, 237 ET Configuration page password, 234 ET discovery - exclude nodes, 236 ET discovery accuracy, 229 ET discovery performance, 229 ET Manual zoning, 226 ET single/incremental node discovery, 231 ET topology filters, 86 etc/hosts, 25 etc/netmask, 117 etc\networks, 29 Ethereal, 180 Etherpage, 172 etrestart.ovpl, 11 ETsNoSnmpNodes.ovpl, 239 event, 98 event – default event, 114 Event configuration, 99 Event configuration Special Characters, 103 event correlation, 123 Event Correlation, 123 event correlation - summary, 133 event correlation and notifications, 166 event data, 262 Event data export filtering, 273 Event Forwarding, 108 Event logging, 104 event Source External File, 168 Event Variables, 103 eventdb - clear, 274 eventdb - dumping, 107 eventdb - queries, 107 eventdb Increase size, 273 Events - Additional Actions, 111 Events - Dump, 105 events - Launch custom URL’s, 111 Events - NNM-generated, 109 Events OpenView Enterprise, 112 Events to Windows Event, 151 EventsBySeverity.ovpl, 107 evntcmd, 151 314 evntwin, 151 Excel, 268 Exchange, 170 excludeip.conf, 29 Exclusion filter, 278 Expect, 182 exponentially-decayed moving average, 195 Extended Topology, 225, 240 Externally manage objects, 43 Extreme filtering in data exports, 273 Failover filters, 277 Fatal IPMAP, 203 Fault Analysis Tools, 175 fbackup, 285 FDDI, 159 filter expressions, 278 Filtering, 277 filters - external files, 278 filters and wildcards, 278 Find - map operation, 199 firewall, 293 Firewalls, 291 Force or restart an ET discovery, 11 Fragment, 162 Frame, 161 frame relay, 55 frecover, 285 frestore, 285 FRF, 7 fuser, 210 Gambit’s MIMIC, 141 General security, 292 generalConnsEdits, 234 getConnectedPort.sh, 270 gethost, 37 gethostbyaddr, 33, 36 Gethostbyaddr, 25 gethostbyname, 33 global.conf, 270 graphs, 160 HC, 159 hexadecimal IP Address conversion, 191 High Availablity, 291 High Capacity, 159 high speed links, 159 Hiplink, 172 H-Node, 34 homebase, 198 Index HomeBase, 240 HOST-RESOURCES, 150 Hosts files, 25 hosts.nnm, 233 hot standby, 285 How node objects are labeled, 27 How node objects are named, 25 HP IT Resource Center, 16 HPoid2type, 7 HSRP, 49, 81, 118, 253 HSRP Status, 252, 253 HSRP View, 252, 253 htgroup, 220, 223 I2Trace, 183 IANA, 114 ICMP, 55, 292, 293 ICMP Burst polls, 66 ICMP Redirects, 66 If%util, 154 IF_Down, 129 ifAlias, 28, 29, 52, 65, 79, 97, 119, 193 ifAlias MIB, 190 ifHCInOctets, 159 ifName, 119 ifNumberQueryEnable, 233 IfTypeFilter, 90 ifTypes, 91 ImageMagik, 160, 221 Important Node filter, 128 Important Node filters, 277 InCharge, 183 Inconsistent Subnet Mask event, 116 Integration Packages, 310 Interface index remapping, 193 interface list for all nodes, 274 interface objects, 28 Interface utilization, 153 interface View, 250 Intermittent Status event correlation, 132 Invalid broadcast IP Address, 27 ioscan, 180 IP address table, 146 IP Addresses Wildcards in event source, 168 IP Hostname, 25 ipAddrTable, 293 ipconfig, 35 ipconfig /flushdns, 35 IPMAP, 207, 208, 284 IPMAP Symbol Changes, 205 IPMAP tracing, 211 ipNoLookup.conf, 29 IPv6, 256 IPv6 Status events, 257 IPv6.conf, 256 IPX address, 26 isDiscoContrivedIF, 85 ISDN, 213 Jakarta, 218 JAVA, 24 JAVA console, 259 JAVA heap size, 258 Java Plugin, 24 Java Plug-in Cache, 259 JAVA programming interface, 182 JAVA_HOME, 24 JAVA-based telnet, 222 javaGrapher, 222 Jovw menu registration, 221 jovw registration file, 223 Jovw registration files, 220 JRE, 19 JVM, 222 Kardinia, 184 kermit, 173 keyboard and mouse, 220 Large object database, 214 launcher, 223 Launcher, 221 LD_LIBRARY_PATH, 18 level 2 discovery, 41 Level 3 paths, 175 level 3 topology, 198 Licensing issues, 17 Limiting Discovery, 41 Limiting menu bar access, 214 Link Down, 112 Link Up, 112 linkDown trap - cisco, 192 LLa address, 26 lmhosts, 34 LMhosts, 25 loadhosts, 48, 49 loadOnlyPolledObjectsIntoMemory, 94 Local Registration Files, 5 localtime, 163 log files, 13 logec, 183 315 Fognet’s Field Guide to OpenView NNM logger, 149 Logging NNM errors to Windows event log, 13 logindevperms, 171 loopback, 25 Lowest numbered IP Address, 26 MAC Address, 26 MAC addresses, 177 mail, 170 mailx, 170 Maintain/preen web reports, 272 Manage/unmanage objects, 205 Managing Systems using NNM, 148 Managing VLANs, 212 Map - Delete From This Submap, 202 Map - Forcing icons to scale, 210 Map - lining up symbols, 209 Map Application Plane, 200 Map cutting and pasting objects, 198 Map Edit-Add Object, 204 Map Executable, 200 Map Export/Import, 284 Map filters, 277 Map find operation, 199 Map Highlighted, 200 Map IP Map” attributes, 204 map Layout, 200 Map Object Visual Cues, 200 Map Operations, 198 map Overlay, 200 map ownership and permissions, 215 Map Selected, 200 Map selecting multiple objects, 198 Map status progagation rules, 206 map status propagation rules, 206 map Submap persistence, 200 Map Transparent, 200 Map User Plane, 200 map Window Geometry, 200 map.lock, 210 MAPISEND, 170 maxMallocPercent, 160 MC ServiceGuard, 288 menu - limiting access, 214 Mercury Interactive, 1 metacharacters, 166 MgmtAddrInhibited, 96 MgmtAddrMaxSnmpQueries, 96 MIB Application Builder, 6, 260 316 MIB Appls in Dynamic Views, 260 MIB Expression Guru, 156 MIB Expressions, 155 mibExpr.conf, 153, 155 mibtable, 179 Microsoft Excel, 268 migratable IP addresses, 288, 289 migrate70to701ZoneNumber.ovpl, 255 migrateHsrpVirtualIP, 50, 251 Mimic SNMP Simulator, 183 mineETDB.ovpl, 270 Missing charsets in String, 22 MLT.pdf, 258 modem, 173 Motif GUI, 198 mplay32, 172 MRTG, 184 MTU, 161 MultiLink PPP, 50 multilinked routers, 49 Multiple copies of NNM, 275 Multiple reboots event correlation, 133 multiple versions of JAVA, 24 Multi-Router Automatic Protection Switching, 50 MyHostID.xml, 93 naa.cnf, 141 naaagt, 141 Nagios, 183 Name lookup performance, 33 Name Services, 25 Name services performing poorly, 32 NameSpace.conf, 133 NameSrvQueryCount, 33 NameSrvQueryTimeout, 33 Naming restrictions, 28, 29 NAT, 293 natping, 177 NDBM, 262 ndd, 178 Neighbor View, 251 Netbios, 34 NetBT, 34 netcheck, 179 netfmt, 180 netmon, 39 netmon -c <critical-route-seedfile>, 61 netmon configuration files, 51 netmon discovery polling, 58, 59 Index netmon -k ConnectorL2Ports, 60 netmon Layer 2 Polling, 59 netmon LRF switch settings, 51, 58 netmon polling statistics, 68 netmon SNMP queries, 52 netmon SNMP-based polls, 60 netmon tracing, 68, 70 Netmon vs. APA, 67 netmon.cmstr, 45, 143 netmon.equivPorts, 202 netmon.MACnoDiscover, 56 netmon.noDiscover, 42, 288 Netmon.noDiscover, 56 netmon.noMACdiscover, 42 netmon.snmpStatus, 61 netmon-loadhosts, 48 netscape, 223 Netscape, 21 NetScout, 186 Net-SNMP, 152, 183 netstat, 178 nettl, 180 nettune, 178 network name, 29 Network named “10”, 211 network objects, 28 network symbols, 211 NIS, 36 NMActiveRoute.Conf, 253 Nmap, 180 nmdemandpoll, 61, 276 nmdemandpoll.ports, 276 NNM 6.2 default data collections, 269 NNM 8i, 295 NNM and multiple NIC cards, 289 NNM APIs, 182 NNM Collection Station, 275 NNM Developer Kit, 182 NNM Filters, 277 NNM Management Station, 275 NNM Patch information, 15 NNM Ports used, 291 NNM Product Documentation, 14 NNM Product Numbers, 311 NNM SPI, 308 NNM UUID, 26 NNM Web Interface, 218 NNM.spec, 261 nnmRptPresenter.exe, 266 No format in trapd.conf, 114 NO_EXTREME_EVENT_FILTERING, 273 Node definition, 307 Node Status, 65 Node status events, 62 Node View, 249, 250 Node_Unknown, 129 Node_Up events, 62 NodeIf event correlation, 129 nodeView, 241 noIpLookup.conf, 36 nolookupdb, 262 non-IP interfaces, 44 Non-IP unconnected port status, 60 nonsnmpnodes.nnm, 236 Nortel MLT and SMLT, 80 NOTIFCATION-TYPE, 114 notification and paging tools, 172 Notifications, 164 NOTIFICATION-TYPE macro, 98, 99 npprobe.log, 176 nscd, 36 nslookup, 35 nsswitch.conf, 35 NTLM, 23 NU0 interfaces, 211 Null interfaces, 211 OAD, 254 Object filters, 277 Object Status, 206 Object-based polling, 58 Octet, 161 ODBC, 268, 272 ODBC Drivers, 263 OID_ALIAS, 192 oid_to_type, 7, 41, 47, 114 Omniback, 282 on-demand submap, 208 oneXmlFileCreator.ovpl, 242 open2, 174 OpenNMS, 183 OpenView Data Protector, 282 OpenView enterprise, 109 OpenView Operations, 148 operational databases, 262 Operational Status Colors, 109 Opnet, 183 Oracle as DW, 267 orphaned objects, 20 317 Fognet’s Field Guide to OpenView NNM OSPF, 26, 252 OSPF View, 252 Outlook, 170 OV poller plus event correlation, 132 ov.conf, 38, 143, 288, 289 ov.envvars, 4 OV_CONF, 4 OV_EVENT, 98, 109 OV_Node_Down, 130 ovAccessPath.ovpl, 176 ovactiond, 103, 164, 165 ovactiond LRF settings, 167 ovaddobj, 5 ovalarmsrv, 110, 111 ovas, 240, 244 ovas not running, 18, 19 ovautoifmgr, 43, 212 ovbackup.ovpl, 281 ovcapsd, 151 ovcoldelsql, 266 ovcolqsql, 266 ovcoltosql, 157, 266 ovcomposer, 124 ovdbcheck, 266 ovdbdebug, 266 ovdbsetup, 266 ovdelobj, 6 ovdumpevents, 104, 105, 107 ovdupip, 255 ovdvstylereset, 222 ovdwconfig.ovpl, 266, 269, 272 ovdwevent, 266, 271 ovdweventflt, 266, 273 ovdwloader, 266 ovdwquery, 107, 263, 264 ovdwtopo, 266 ovdwtrend, 266, 271 ovdwunloader, 266 Overlapping Address Spaces, 254, 255 OverTime, 184 ovet_apaConfig.ovpl, 39, 74 ovet_bridge, 39 ovet_da* not running, 21 ovet_daHSRPSnmp.log, 254 ovet_demandpoll.ovpl, 75 ovet_dhsnmp.log, 228 ovet_disco, 39 ovet_fixTopology.ovpl, 238 ovet_generateTopoDeltaReport.ovpl, 238 318 ovet_pathengine, 128 ovet_poll, 39 ovet_poll startup time, 230 ovet_reloadTopoDBTbls.ovpl, 238 ovet_topoconndump.ovpl, 235, 270 ovet_topoconnedit.ovpl, 235, 237 ovet_topodump.ovpl, 78, 86, 238 ovet_topoobjcount.ovpl, 238 ovet_topoquery, 235, 238, 253 ovet_toposet.ovpl, 78 ovet_truncatetopotbls_all.ovpl, 239 ovevent, 102 ovexprguru, 156 ovfiltercheck, 277 ovfiltertest, 277 OVHIDESHELL, 169 ovhtpasswd, 220 OVO, 126, 148 ovobjprint, 7, 40 ovotopodump, 307, 308 ovpause, 281 ovrequestd, 262, 268 ovrestore.ovpl, 281 ovresume, 281 ovserver file, 216 OVSHELL, 169 ovsnmp.conf, 262 ovspmd.auth, 216 ovstart, 13, 23 ovstart.allow, 13 ovstatus, 23, 210 ovstop, 211 ovtopodbsnapshot, 285 ovtopodbsnapshot.ovpl, 238 ovtopodump, 20, 40, 203, 274, 277, 280, 301 ovtopofix, 43, 203 ovtrapd, 98, 100, 109, 145 ovw, 198, 200, 215 ovw -fields, 9 ovw – list open maps, 210 ovw map ownership and permissions, 215 ovw map sessions for web GUI, 219 ovw –mapcount, 203 ovw maps – limiting access, 215 ovw symbol status, 207 ovw –verify, 9 OVw*layoutMinSymbol Radius, 210 ovw.auth, 215, 216 ovwchgrp, 215 Index ovwchmod, 215 ovwchown, 215 ovwdb fields, 23 ovwdb.auth, 216 ovweb.conf, 223 ovwinit failed errors, 203 ovwNTtools.exe, 216 ovwperms, 215 OVwRegDir, 214 OVwRegIntro, 6 Packet, 161 Packet Sniffers, 180 paConfig.xml, 83 pager, 173 PairedTimeWindow, 131 PairWise event correlation, 130, 131 password, 3, 220, 234 passwords, 3 pathping, 179 Paths, 2, 3 Payload, 162 PD, 175 pdcentral.sh, 175 PDU, 161 performance and fault graphs, 156 Performance Insight, 300, 309, 317 Perl, 146, 174 PERL, 47 Perl for action callbacks, 168 persistence filter, 208 Persistence filters, 277 physAddr.conf, 178 PhysAddr_Mismatch, 116 pickSnmpAddrPoll, 45 pickSnmpAddrPolls, 143 Pickup, 170 Ping, 165 ping sweep attacks, 292 Ping Tools, 177 pingPercentRetry, 158 pingResponseTime, 158 pipes, 169 pmd, 137 pmd LRF, 104 pmd Performance, 137 pmdmgr, 137, 180 P-Node, 34 PollerPlus.fs, 133 Polling through firewalls, 276 PollingEngineThreadPoolSize, 94 pop-up menu, 216 port 162, 145, 292 port address mappings, 178 Ports used by MS and CS, 276 Postie, 173 Preferred SNMP address, 45, 46 Preferred IP, 26 preferred IP address, 29 preferred SNMP address, 27 Printing graphs (UNIX), 160 Problem Diagnostics, 175, 307, 309 Product Feature Deltas, 304 propagateStatusForUnconnectedObj, 94 proxy-ARP, 116 PTR record, 36 PTR records, 26 Putty, 182 qpage, 173 RAMS, 309 RDMI, 151 read-write maps – limiting access, 215 recVals.conf, 233 redirects, 169 Re-do discovery (start from scratch), 12 refresh Homebase, 11 RegExp in map find, 199 Regular expressions in find, 199 regverify, 6, 9, 217 reject all traps, 121 release history, 303 Remove a stubborn object, 203 REMOVED objects, 204 repackDb.sh, 271 Repeated event correlation, 130 Reporter URLs, 266 Reporting, 262 Reporting tools, 270 reportPresenter, 271 Reports - delete old, 271 request.properties, 262 request_list, 262 resolv.conf, 35 Resource Manager Essentials, 186 Restore, 281 RFC 1027, 116 RFC 1213, 213 RFC 1215, 98 RFC 1253, 252 319 Fognet’s Field Guide to OpenView NNM RFC 1493, 213 RFC 1518, 118 RFC 1573, 192 RFC 1850, 252 rfc 2233, 159 RFC 2233, 192 RFC 2578, 98 rfc 2790, 150 RFC 950, 118 RFC1034, 29 RFC1157, 144 RFC1757, 162 RFC2021, 162 ringBell, 165 ringBell.ovpl, 171 RME, 186 RMON, 162 rnetstat, 178 Route Analytics, 309 Router/Switch Health, 129 routing switches, 213, 237 RPN, 156 RRDtool, 183 rrset-order, 31 Scheduled polls, 128 SEC – Simple Event Correlator, 183 seed file, 45 Seeding Discovery, 44, 45 sendmail, 170 sendMsg, 164 sendMsg.ovpl, 100 sendpage, 173 server.xml, 242 ServiceProvider, 34 setstatus.ovpl, 209 setupExtTopo.ovpl, 39, 225 setupSyslog.ovpl, 149 Smart path, 176 Smart Plug-ins, 308 SMON, 162 SMS, 145 SNMP, 112 SNMP – multiple agents, 140 SNMP address - controlling, 143 SNMP agent, 9 SNMP community string, 90 SNMP community strings, 45, 140 SNMP configuration database, 262 SNMP cut views, 190 320 SNMP Data Collection Terminology, 161 SNMP discovery/polling -stop, 44 snmp dot1dTpFdbTable, 229 SNMP Extensible agents, 152 SNMP ExtensionAgents, 141 SNMP HOST RESOURCES MIB, 150 SNMP manager commands, 145 SNMP Manger and Agent functions, 139 SNMP master agent, 152 SNMP Master Agent, 141 SNMP MIB, 40 SNMP OID, 114 SNMP PDU, 144 SNMP Research, 139, 309 SNMP Security Pack, 139, 309 SNMP sysName, 26 SNMP trap, 98 SNMP trap ports, 145 SNMP trap storms, 101 SNMP traps - configuring, 98 SNMP traps - eliminating, 121 SNMP traps - filter out, 100 SNMP traps - generating, 102 SNMP vendor ID, 114 SNMP versions, 140 SNMP versions and history, 139 SNMP vulnerabilities, 144 SNMP Web Resources, 147 snmpColDump, 163, 270 snmpCollect, 153 snmpCollect commands, 157 snmpCollect statistical thresholds, 157 snmpColStats.txt, 157 snmpnolookupconf, 29, 36 snmpnotify, 100, 102 snmpset, 26, 120 snmptrap, 102 SNMPv2c, 112 SNMPv2U, 140 SNMPv3, 141, 142 SNMPv3 and VLANS, 143 snmpviewer, 221 snoop, 180 software loopback address, 26 softwareLoopback, 25 SolarWinds, 183 Solid DB, 262 Solid DB schema, 264 solid.ini, 263, 265 Index Solidtech, 262 Solistice Enterprise Agents, 152 Sound notifications, 171 SPAN, 160 Spatch, 173 spinning in ifTable, 115 SSH, 217 Stack tracing via pmdmgr, 137 Standard Edition, 307 statTimeRanges.conf, 157 status bridge - disable, 95 status colors, 109 Status event text, 66 Status event variable bindings, 65 Status Polling via netmon, 55 statusAnalyzerThreadPoolSize, 94 submap properties, 200 Sun Cluster, 288 Support Matrix, 304 Support Package Numbers, 313 Swatch, 182 Switching routers, 213, 237 symbol and object delete operations, 202 Symbol Status, 206 Symbol status influences, 207 Symbols superimposed, 212 synchorizing, 202 Syslog integration facility, 148 sysObjectId, 40 System Insight, 162 system performance, 148 system resources, 148 SystemEdge, 152 Systems Insight, 310 Tap Modem Phone numbers, 173 Tavve, 108 Tcpdump, 180 Telalert, 172 Terminal Server, 220 testnonSnmpFile.ovpl, 237 Third Party Tools, 182 time - Epoch, 106 Tivoli NetView, 108 Tomcat, 218 TopN report, 270 topodbpoke, 47 TopoFilters.xml, 86, 230 TopoInventory.ovpl, 270 topology, 262 topology filters, 286 Topology filters, 277 traceroute, 175, 179 tracert.exe, 179 transient submap, 208 Transient, persistent and on-demand submaps, 208 trap, 98 trap destinations, 142 Trap OIDs - Weird, 191 trap source, 114 TrapBlaster, 183 trapd.conf, 99 trapd.log, 104 TRAP-TYPE macro, 98 trend, 262 Troubleshooting Windows Paths, 10 trunked or meshed connections, 202 trustedCmds.conf, 164 UDP, 144 ufsdump, 285 ufsrestore, 285 ULIMIT, 160 Unable to load any useable fontset, 22 UnconnectedEthIFInConnectedNode, 94 undesirable traps, 121 UNIX clusters, 288 unmanage objects via URLs, 258 unmanaged, 20, 201 unmanageSecondary, 276 Unrecognized ASN.1 type events, 121 useIcmpIfSnmpNoSuchObj, 95 useIfAlias=false, 119 userAuthentication.xml, 243 Using monochrome images, 21 uucp, 173 Variable Bindings, 103 variable-length subnet masks, 117 vendor, 8 vendor ID, 115 Veritas Cluster 2.0, 288 versions, 295 Virtual directories, 17 VivIt, 2 VLAN, 188, 212 VLSM, 117 VNC, 183, 219 VPNs, 212 VRRP, 81, 118 321 Fognet’s Field Guide to OpenView NNM VRRP View, 251 WanIf, 90 Warm Start, 112 web browser – setting preferred, 222 Web GUI, 218 Web GUI Access Control, 223, 224 Web GUI login password, 220 Web GUI menu and toolbar - adding, 220 Web interface URLs, 218 web.xml, 242, 243 webappmon, 221 Window Scripting Host, 169 Windows Events to NNM, 151 Windows tracing and logging, 13 WinDump, 180 WINS, 25, 33, 34 winsnmp, 145 Wireless, 214 WMI via SNMP, 152 wpaagt, 141 writeSelList, 270 WshShell, 169 wsnmp32.dll, 145 322 Xdefaults, 22 xfs, 23 XNmevent, 110 xnmevents, 99, 164 xnmeventsExt.conf, 111 xnmgraph, 154, 161 xnmgraph command examples, 160 xnmloadmib, 99 xnmpolling, 39, 42 xnmsnmpconf, 37, 61, 120, 140 xnmsnmpconf -clearC, 35 xnmsnmpconf –clearCache, 203 xnmsnmpconf -export, 283 xnmtopoconf, 205, 279 xnmtrap, 100, 108, 164 xpr, 160 xrdb, 22 xterm, 22, 171 xvfb, 219 xwd, 160 Zone Discovery, 233 zones, 226