Download here - Fognet Consulting

Fognet’s Field Guide to
OpenView
Network Node Manager
2nd Edition
Mike Peckar
Fogbooks®
West Boylston, Massachusetts
Fognet’s Field Guide to OpenView NNM
Fognet’s Field Guide to OpenView Network Node Manager
Second Edition
Mike Peckar
Copyright © 2006, 2008, Fogbooks and Mike Peckar, All Rights Reserved .
Printed in the United States of Am erica, Spain, and the United Kingd om
ISBN 0-9785627-2-4
N o p art of this p u blication may be r ep rod u ced or transmitted by an y m eans, electronic or m echanical,
inclu d ing p hotocop ying, record ing, or any information storage an d retrieval system , w ithou t p erm ission in
w riting from the au thor.
All bran d names and p rod u ct names m entioned in this book are trad em arks or registered trad em arks of their
resp ective comp anies.
Lim it of Liability/ Disclaimer of Warranty:
The p u blisher an d the au thor make no rep resentations or w arranties w ith resp ect to the accu racy or
comp leteness of the contents of this w ork an d sp ecifically d isclaim all w arranties, inclu d in g w ithou t lim itation
w arranties of fitness for a p articu lar p u rp ose. N o w arranty may be created or extend ed by sales or
p romotional m aterials. The ad vice and strategies contained herein may not be su itable for every situ ation. If
p rofessional assistance is requ ired , the services of a comp etent p rofessional p erson shou ld be sou ght. N either
the p u blisher nor the au thor shall be liable for d amages arising herefrom .
The fact that an organization or w ebsite is referred to in this w ork as a citation and / or p otential sou rce of
fu rther in formation d oes not mean that the au thor or p u blisher en d orses the in form ation the organization or
w ebsite may p rovid e or recommen d ations it may make. Fu rther, read ers shou ld be aw are that internet
w ebsites listed in this w ork m ay have changed or d isa p p eared betw een the tim e w hen this w ork w as w ritten
and w hen it is read .
LR21
Pu blisher contact, feed back, and ord ers:
ii
w w w .fogbooks.com
Contents at a Glance
1- Introd u ction ..........................................1
2 - General Information ............................3
3 - Com m on Issu es ..................................16
4 - N N M and Nam e Services .................25
5 - Discovery .............................................39
6 - Status Polling - netmon ......................55
7 - Status Polling - APA ..........................71
8 - Traps, Events and Alarm s ................98
9 - Interpreting Events ..........................112
10 - Event Correlation .............................123
11 - SN MP Fu nctions ..............................139
12 - System s Management ......................148
13 - Data Collection and Threshold s ....153
14 - N otifications ......................................164
15 - Fau lt Analysis Tools ........................175
16 - Third Party Tools .............................182
17 - Cisco Devices ....................................185
18 - ovw Map Operations ........................198
19 - Web Interface ....................................218
20 - Extend ed Topology ..........................225
21 - Dynam ic View s ................................240
22 - Databases and Reporting ................262
23 - Distribu ted N NM .............................275
24 - H igh Availability and Backu p .......281
25 - Firew alls ............................................291
26 - Prod u ct Details .................................295
Glossary .......................................................314
Ind ex .............................................................320
iii
Table of Contents
Contents at a Glance ................................ iii
Table of Contents ...................................... iv
1. Introduction ............................................ 1
Acknow led gem ents .............................................................................. 2
About Fognet and the author .............................................................. 2
Conventions used in this guid e .......................................................... 2
2. General Information ............................. 3
Paths ....................................................................................................... 3
Default passw ord s ................................................................................ 3
OpenView environm ent variables ..................................................... 4
Extend ing OV environment variables ............................................... 5
N N M environm ent variables usage restrictions .............................. 5
Custom ize d aem ons via local registration files (LRF) ..................... 5
Application registration files (ARF) ................................................... 6
Sym bols, OID types and field s (FRF) ................................................ 7
Application d efaults upd ate proced ure (app -d efaults) .................. 9
Troubleshooting Wind ow s paths ..................................................... 10
Force or restart an ET d iscovery ....................................................... 10
Re-d o d iscovery (start from scratch) ................................................ 12
Logging N N M errors to Wind ow s event log .................................. 12
Running ovstart and ovpause as non -root (UN IX) ........................ 12
Com m on log files ................................................................................ 13
N N M prod uct d ocum entation .......................................................... 13
Man Pages (Linux) .............................................................................. 14
N N M patch inform ation .................................................................... 14
iv
Table of Contents
3. Common Issues .................................... 15
Troubleshooting resources ................................................................. 15
Licensing issues d ue to upgrad es, m igrations, hom ing ................ 16
Changing N N M server hostnam e or IP ad d ress ............................ 16
Installation issue: virtual d irectories not created ............................ 16
Installation issue: ovas not running ................................................... 17
Installation issue (UN IX): Using N FS m ounted CDs ..................... 17
Installation issue (Wind ow s): N on-stand ard directory ................. 18
Installation issue (Wind ow s): Term inal Services ............................ 18
Unable to d ow nload JRE through firew all ...................................... 18
Devices discovered in unm anaged state .......................................... 18
Cleaning orphaned objects from object d atabase ........................... 19
Daem ons ovet_d a* not running (ovstatus) ...................................... 19
N etscape and N N M co-existence (UN IX) ........................................ 20
Cannot allocate 128 colors - using m onochrom e im ages ............... 20
Unable to load any useable font set (UN IX) .................................... 20
xterm : unable to locate a suitable font (UN IX) ............................... 21
Missing charsets in string to fontset (Solaris).................................. 21
Missing charsets in string to fontset (H P-UX)................................. 21
N TLM - passw ord need ed - firew all: unknow n ............................. 21
Troubleshooting d aem ons using ovstart and ovstatus .................. 22
Autom atically d etect and restart OV d aem ons that d ie ................ 22
Filtering on custom ovwdb field s ....................................................... 22
Working w ith m ultiple JAVA versions ............................................ 22
4. NNM and Name Services ................... 24
H ow nod e objects are nam ed ............................................................ 24
H ow preferred IP ad d ress for SN MP is chosen .............................. 26
H ow nod e objects are labeled ............................................................ 26
H ow interface objects are nam ed by netmon ................................... 27
ifDescr nam ing restrictions ................................................................ 27
v
Fognet’s Field Guide to OpenView NNM
H ow netw ork objects are nam ed ...................................................... 27
N od e and interface nam ing restrictions .......................................... 28
ipN oLookup.conf, snm pnolookupconf, exclud eip.conf ............... 28
General DN S consid erations ............................................................. 28
Cache-only DN S.................................................................................. 29
Split horizon DN S............................................................................... 30
BIN D-based DN S im plem entation troubleshooting ...................... 30
BIN D-based DN S on Wind ow s ........................................................ 31
“N am e services are perform ing poorly” alarm s ............................ 31
Tuning lookup perform ance (Wind ow s) ........................................ 32
Tuning lookup perform ance (UN IX) ............................................... 33
N N M d oesn‟t d etect DNS changes (Wind ow s) .............................. 34
To A record or not to A record ......................................................... 34
A record com pare issue on Solaris platform ................................... 35
nscd issues on Solaris .......................................................................... 35
N am e resolution support com m and s: ............................................. 35
N N M DN S tracing .............................................................................. 36
N am e services-related docum entation ............................................ 36
N N M in d em o, DH CP or m obile environm ents ............................ 36
5. Discovery via netmon .......................... 37
Preparing for N N M d iscovery .......................................................... 38
After N N M discovery ........................................................................ 38
Discovery process in a nutshell ........................................................ 38
Discovery polls vs. other types of netmon polls .............................. 39
Und erstanding netmon-based level 2 d iscovery ............................. 39
Lim iting d iscovery .............................................................................. 39
Externally m anaging or unm anaging objects ................................. 41
Stop d iscovery of non -IP interfaces for a d evice class ................... 42
Stop all SN MP discovery/ pollin g to a d evice class ....................... 42
Discovery hints ................................................................................... 42
Seed ing d iscovery ............................................................................... 42
vi
Table of Contents
Configuring m ultiple SNMP com m unity strings ........................... 43
Preferred SN MP m anagem ent ad d ress ............................................ 43
Forcing the SN MP ad d ress for a nod e ............................................. 44
netmon‟s auto-adjusting d iscovery polling algorithm .................... 45
PERL script: find OIDs not listed in oid _to_type ........................... 45
load hosts ............................................................................................... 45
Discovery of 31 and 32 bit subnetw orks .......................................... 47
Discovery of H SRP, clusters, and m ultilinked routers .................. 47
Sum m ary of netmon configuration files ........................................... 48
Und ocum ented netmon LRF sw itch settings ................................... 49
SN MP queries issued by netmon ....................................................... 49
6. Status Polling - netmon ....................... 52
ICMP and SN MP id iosyncrasies and “brow n -outs” ...................... 52
Ad justing ICMP and SNMP status polling intervals ..................... 52
netm on.noDiscover and netm on.MACnoDiscover files................ 53
netm on.interfaceN oDiscover file ...................................................... 53
Com m only used netmon LRF sw itch settings .................................. 55
netmon global status polling d efault ................................................. 55
netmon object-based polling (V6.2+) ................................................. 55
Dynam ically-adjusting status polling by netmon ............................ 55
netmon layer 2 polling algorithm s ..................................................... 56
N on-IP unconnected port status (netmon) ....................................... 57
netmon SN MP-based status for nod es .............................................. 57
Determ ine netmon-based polling intervals for a d evice ................. 58
netmon critical path analysis .............................................................. 58
N od e status events – interpretation .................................................. 59
H and ling m ultiple OV_N od e_Up events ........................................ 59
Status event variable bind ings of interest ........................................ 62
ICMP burst polls ................................................................................. 63
ICMP red irects ..................................................................................... 63
netmon vs. APA poller ........................................................................ 64
vii
Fognet’s Field Guide to OpenView NNM
netmon polling statistics ..................................................................... 65
netmon troubleshooting, tracing and d um ping .............................. 67
7. Status Polling - APA............................ 68
APA Architecture ............................................................................... 68
APA vs netmon status architecture ................................................... 69
APA and Sm art Plug-Ins ................................................................... 71
Turn APA polling on or off (“the big sw itch”) ............................... 71
Determ ine poller control.................................................................... 71
APA configuration in a nutshell ....................................................... 71
Dem and polling using ovet_d em and poll.ovpl .............................. 72
APA Island Group Monitoring ......................................................... 73
APA status events ............................................................................... 74
APA status from SN MP traps ........................................................... 74
Suppress/ allow APA status polling ................................................ 75
APA status events ............................................................................... 75
APA status as reflected in ovw m aps .............................................. 76
APA aggregated port support .......................................................... 77
APA H SRP and VRRP status support ............................................. 78
APA board status support ................................................................. 79
paConfig.xm l: the APA configuration file ...................................... 80
paConfig.xm l evaluation ord er issues ............................................. 81
Interface ICMP polling ....................................................................... 81
Synchronizing poll settings w ith configuration polls ................... 81
paConfig.xm l polling granularity class specifications .................. 81
Using ET topology filters to specify polling granularity .............. 82
Filter assertion type attributes .......................................................... 83
Filter nod es based on SNMP sysObjectID ....................................... 85
Force a d evice to be polled via ICMP or SN MP only .................... 86
Filtering by ifType (APA) .................................................................. 87
Disable ICMP to a firew all (APA) .................................................... 87
Sw itching routers and routing sw itches in the APA ..................... 88
viii
Table of Contents
Interesting sw itch interface filter exam ple ...................................... 88
APA topology events .......................................................................... 89
APA and im portant node filters ....................................................... 89
APA perform ance im provem ents ..................................................... 90
Connected vs unconnected interface APA status ........................... 90
APA m em ory footprint red uction .................................................... 91
Disabling status brid ge ....................................................................... 91
Characterizing APA polling behavior .............................................. 91
Im proving SN MP-based status w ith the APA ................................ 91
APA and Managem entAd d esss picking .......................................... 92
Troubleshooting APA ......................................................................... 93
8. Traps, Events and Alarms .................. 94
What is a trap vs. an event vs. an alarm ? ......................................... 94
Configuring SN MP traps via trap m acros ....................................... 94
Configuring events or traps via trap d efinition files ...................... 95
Configuring events or traps m anually ............................................. 95
Drop SN MP traps from particular d evices ...................................... 96
Autom atically suppress SN MP trap storm s .................................... 97
Generating ad -hoc SN MP traps or N N M events ............................ 98
Differences betw een snmpnotify and ovevent ............................... 98
Event variable bind ings ...................................................................... 99
Event logging ..................................................................................... 100
Dum ping the entire eventdb ............................................................. 101
Interpreting ovd um pevents output ................................................ 101
Dum ping parts of eventdb................................................................. 103
Ad hoc queries of the eventdb .......................................................... 103
Trap and event forw ard ing .............................................................. 103
OpenView enterprise and N N M-generated events ..................... 104
Alarm and icon status color d efaults .............................................. 105
ovalarmsrv LRF settings .................................................................... 106
Alarm brow ser settings (XN m event app -d efaults) ...................... 106
ix
Fognet’s Field Guide to OpenView NNM
Ad d itional actions ............................................................................ 106
9. Interpreting Events ............................ 107
N N M‟s hand ling of generic and specific traps ............................. 107
Generic traps and origin event OIDs ............................................. 107
“Received event X, N o form at in trapd .conf” ............................... 108
Agent in d istress; “spinning in ifTable or ipAd d rTable” ........... 110
OV_PhysAd d r_Mism atch ............................................................... 110
Inconsistent subnet mask................................................................. 111
OV_Duplicate_IP_Ad d ress ............................................................. 113
OV_DuplicateIfAlias ........................................................................ 113
Authentication failures .................................................................... 114
Unknow n or unrecognized ASN .1 type # m essages ................... 115
Elim inating events from und esirable system s .............................. 116
10. Event Correlation .............................. 117
What is event correlation? ............................................................... 117
Correlation Manager ........................................................................ 118
Correlation Com poser ...................................................................... 118
ECS Designer ..................................................................................... 120
Instances and stream s ...................................................................... 120
Special correlations ........................................................................... 121
Connector d ow n event correlation circuit .................................... 121
N od eIf event correlation .................................................................. 123
Repeated event correlation circuit .................................................. 124
PairWise event correlation circuit .................................................. 124
DeDup event correlation ................................................................. 125
Interm ittent Status event correlation ............................................. 125
OV poller plus event correlation .................................................... 126
Multiple reboots event correlation ................................................. 126
Cisco chassis event correlation ....................................................... 127
x
Table of Contents
Sum m ary of event correlations affecting status........................... 127
Useful ECS m anager com m and s ..................................................... 129
Configuring ECS circuits .................................................................. 129
Disable or enable ECS ....................................................................... 129
Logging incoming ECS events ........................................................ 130
Logging output and correlated events ........................................... 130
Sim ulate events for testing ECS logic ............................................. 130
ECS tracing ......................................................................................... 131
Stack tracing via pm d m gr ................................................................ 131
Monitoring pmd bottlenecks (N N M 6.1 only) ............................... 131
Correlation Com poser tracing ......................................................... 131
11. SNMP Functions ............................... 133
SN MP versions and history ............................................................. 133
Configuring SN MP comm unity strings on N N M agent ............. 134
Configuring SN MP versions ........................................................... 134
Configuring m ultiple SNMP agents on a single nod e ................. 134
SN MP m aster agent sw itches .......................................................... 135
SN MPv3 configuration using SN MP security pack ..................... 135
SN MPv3 Security Pack installation ................................................ 136
Configuring and controlling trap d estinations ............................. 136
SN MPv3 and VLAN inform ation ................................................... 137
Controlling the N N M server SN MP ad d ress ................................ 137
Controlling an agent‟s SN MP ad d ress ........................................... 137
Com m unity string d iscovery ........................................................... 137
CERT ad visory CA-2002-03, SN MP vulnerabilities ..................... 138
SN MP PDU size lim itations ............................................................. 138
ovtrapd d aem on not starting (Wind ow s) ........................................ 139
N N M w ith SMS or other SN MP tools (Wind ow s) ....................... 139
Bind ing SN MP trap reception to ports other than 162 ................ 139
SN MP m anager com m and line utilities ......................................... 139
Strip OID and type data from SN MP query output ..................... 140
xi
Fognet’s Field Guide to OpenView NNM
Perl script: output a node‟s IP ad d ress table via SN MP.............. 140
SN MP w eb resources ....................................................................... 141
12. Systems Management ...................... 142
Syslog integration facility ................................................................ 142
Managing system s w ith H OST RESOURCES MIB ...................... 144
Monitoring processes using H OST RESOURCES MIB ............... 144
Monitoring d isks using H OST RESOURCES MIB ....................... 144
Managing system s using RDMI (Wind ow s) ................................. 144
Forw ard ing Wind ow s events to N N M as SN MP traps............... 145
Forw ard ing N N M events to Wind ow s event log ........................ 145
Managing system s using Solstice enterprise agents (SEA )......... 145
Accessing WMI d ata via SN MP ...................................................... 146
N N M server system m anagem ent via SN MP (UN IX) ................ 146
Extensible SN MP agents .................................................................. 146
13. Data Collection and Thresholds .... 147
Data collection best practices .......................................................... 148
If%util (or other % expression) is greater that 100% .................... 148
MIB expressions using mibExpr.conf and mib.coerce ................ 149
MIB expression guru (ovexprguru) ............................................... 150
Und erstanding and m anipulating statistical threshold s............. 151
snmpCollect com m and s ..................................................................... 151
pingResponseTim e, p ingPercentRetry threshold events ............ 151
Data collection and d evice naming/ IP ad d ress issues ................ 152
OpenView specifics reserved for custom events .......................... 152
Data collection and high speed links ............................................. 152
Exhausted 32-bit counters w hen 64-bit is supported .................. 153
Printing graphs (UN IX) ................................................................... 153
Insufficient mem ory w hen launching graphs (Solaris) ............... 154
Exporting graphs to other im age form ats (UN IX) ....................... 154
xii
Table of Contents
Data collections on VLAN s .............................................................. 154
xnm graph com mand exam ples ....................................................... 154
SN MP d ata collection term inology ................................................. 155
System Insight (Com paq Insight) Manager MIBs ........................ 155
SN MP d ata collection vs. RMON .................................................... 156
Calculate epoch tim e for snm pColDum p, etc. .............................. 156
14. Notifications ..................................... 157
Configuring an action callback........................................................ 157
Using trusted Cm d s.conf file ............................................................ 157
General action callback usage consid erations ............................... 158
Metacharacters in action callbacks.................................................. 159
Selected special variables in action callbacks ................................ 159
N otification interactions w ith event correlation ........................... 159
ovactiond LRF settings ....................................................................... 160
Lim iting Action callbacks to a subset of objects ........................... 160
External File to specify action callback sources ............................ 161
Using Perl for actions callbacks ....................................................... 161
Action callbacks on Wind ow s platform s ....................................... 161
Email notifications using native tools (Wind ow s) ....................... 162
Email notifications using native tools (UN IX) .............................. 163
Email notifications and DN S issues (UN IX) .................................. 163
Aud io notifications (UN IX) ............................................................. 164
Sound notifications (Wind ow s)....................................................... 164
Exam ple notification setups............................................................. 164
Popu lar notification and paging tools ............................................ 165
Configuring an attached m od em on UN IX servers ..................... 166
15. Fault Analysis Tools ......................... 167
Problem diagnostics (PD)................................................................. 167
Sm art path (access path view ) ......................................................... 168
Find rouge MAC ad d resses ............................................................. 169
xiii
Fognet’s Field Guide to OpenView NNM
The ping tools .................................................................................... 169
Isolating local netw ork issues ......................................................... 170
rnetstat ................................................................................................ 170
Display port ad d ress m appings ..................................................... 170
netcheck.............................................................................................. 171
m ibtable .............................................................................................. 171
tracert and traceroute ....................................................................... 171
Packet sniffers ................................................................................... 172
Using nettl.......................................................................................... 172
16. Third Party Tools .............................. 174
Environm ental m onitoring and contact management ................ 174
Freew are utilities for UNIX ............................................................. 174
Emulating X w ind ow s on Wind ow s m achines ............................ 175
Event correlation tools ..................................................................... 175
SN MP and netw ork m anagem ent tools ........................................ 175
Topology and netw ork m od eling tools ......................................... 175
Reporting and graphing tools ......................................................... 175
17. Cisco Devices ..................................... 177
Cisco MIBs ......................................................................................... 177
CiscoWorks and N N M ..................................................................... 178
CiscoWorks, N N M, JPI and Internet Explorer ............................. 178
N N M and handling Cisco VLAN s ................................................. 179
N N M‟s Cisco discovery configuration feature ............................. 180
Setting ET to prefer CDP over FDB connection data ................... 181
Useful Cisco IOS SN MP com m and s .............................................. 181
Ad vanced Board status via the APA ............................................. 182
Com m and to create cut SN MP view s ............................................ 182
H ex target IP for Cisco ping MIB ................................................... 182
Und ocum ented IOS comm and s ..................................................... 183
xiv
Table of Contents
Weird trap OIDs (.1.3.6.1.6.3.1.1.5.4.1.3.6.1.4.1.9) ......................... 183
Unknow n trap 1.3.6.1.2.1.0.1&2 from Catalyst 2950 .................... 183
Cisco linkDow n trap configuration ................................................ 184
Controlling Cisco link d ow n trap variable bind ings ................... 184
Interface ind ex rem apping ............................................................... 185
ifSpeed and polling issues w ith d ata collections .......................... 185
Interesting Cisco MIBS, OIDs and MibExprs ................................ 185
What is exponentially-d ecayed m oving average? ........................ 187
Cisco tem perature probes ................................................................ 187
Directly integrating Cisco syslog m essages................................... 187
Cisco-specific event correlation circuits in N N M ......................... 188
Cisco links .......................................................................................... 188
CiscoWorks links ............................................................................... 189
18. ovw Map Operations........................ 190
Map operations – selecting, cutting and pasting objects ............. 190
Tips for using the find operation .................................................... 191
Setting unset capability field s .......................................................... 191
Map object visual cues ...................................................................... 192
Sum m ary of ovw subm ap properties .............................................. 192
N ew objects d o not show up in m aps ............................................ 193
Tw o d evices com bine into a single nod e ....................................... 193
Controlling trunked or m eshed connections in ovw .................... 194
Und erstanding sym bol and object d elete operations .................. 194
Fatal IPMAP or ovw init failed errors ............................................. 195
Rem ove a stubborn object from all maps and d atabases............. 195
Rem ove “REMOVED” objects from d atabases ............................. 195
Manually ad d objects ........................................................................ 196
Externally m anage or unm anage objects ....................................... 197
Overrid ing IPMAP sym bol changes ............................................... 197
Map status propagation rules .......................................................... 198
Sym bol status ..................................................................................... 198
xv
Fognet’s Field Guide to OpenView NNM
Transient, persistent and on -d em and submaps ........................... 200
setStatus utility .................................................................................. 200
Lining up sym bols in a perfect line or row ................................... 201
Forcing icons to scale d ow n in size ................................................ 201
Forcing icons to scale up in size...................................................... 202
Find ing and closing open ovw m aps .............................................. 202
IPMAP tracing ................................................................................... 202
N etw ork nam ed “10” or “arpanet” on internet subm ap ............ 203
N U0 interfaces turn red in m ap ...................................................... 203
Sym bols superim posed on each other in subm ap ....................... 203
Managing VLAN s ............................................................................. 204
Managing VPN end points ............................................................... 204
Managing ISDN interfaces .............................................................. 205
Sw itching routers and routing sw itches in ovw ............................ 205
Wireless .............................................................................................. 206
Tuning ovwdb for large num bers of objects ................................... 206
Lim iting ovw m enu bar access for som e users .............................. 206
Lim iting usage of read -w rite m aps ................................................ 206
Lim iting access to certain N N M applications ............................... 207
Manipulating m ap ow nership and perm issions .......................... 207
ovw .auth, ovw db.auth, ovspm d .auth, and ovserver files .......... 207
Wind ow s-specific m onitoring tools in N N M Menus .................. 208
Ad d right-click pop-up m enu item s to ovw GUI .......................... 208
Ad d m enu item to launch SSH on selection (UN IX) ................... 208
19. Web Interface .................................... 210
URLs ................................................................................................... 210
Maintaining open ovw map sessions for w eb clients ................... 211
Web GUI login passw ord ................................................................ 212
Web GUI access using jovw registration files ............................... 212
Ad d ing Web GUI m enu and toolbar item s ................................... 212
Icons OK in ovw but not OK in w eb GUI ...................................... 213
xvi
Table of Contents
javaGrapher contributed d ata collection tool ................................ 214
Running JAVA apps in event action callbacks ............................. 214
Secure JAVA-based telnet or ssh ..................................................... 214
Setting the preferred w eb brow ser ................................................. 214
Web GUI access control .................................................................... 215
Web GUI and ovw auth files ........................................................... 216
20. Extended Topology........................... 217
Enabling Extend ed Topology .......................................................... 217
ET autozoning .................................................................................... 218
Manual zoning tips ........................................................................... 218
Restart or force ET d iscovery........................................................... 219
ET d evice agents ................................................................................ 219
ET d iscovery and SN MP configuration ......................................... 219
ET d iscovery consid erations and limitations ................................ 220
Im proving ET d iscovery accuracy .................................................. 221
Im proving ET d iscovery perform ance ........................................... 221
Red ucing ovet_poll startup tim e ..................................................... 222
ET and ovw d atabase synchronization issues ................................ 223
ET single/ increm ental nod e d iscovery .......................................... 223
Detect interface configuration changes .......................................... 224
Detect interface table add itions ....................................................... 225
Zone d iscovery tips and perform ance tricks ................................. 225
Passw ord to access ET Configuration page ................................... 226
Connection ed itor .............................................................................. 226
End nod es not in sam e zone as connector ..................................... 227
Ad d connections for unsupported d evices ................................... 227
Layer 3 ed ge connectivity ................................................................ 228
Exclud e nod es from ET d iscovery .................................................. 228
Show ing d evice d etails for non -SN MP d evices ............................ 228
Sw itching routers and routing sw itches in ET .............................. 229
ET com m and sum m ary, support tools, log files ........................... 229
xvii
Fognet’s Field Guide to OpenView NNM
ET and DN S issues ........................................................................... 231
21. Dynamic Views ................................. 232
Dynam ic view s available ................................................................. 232
Dynam ic View s URLs ...................................................................... 233
Dynam ic View s configuration ........................................................ 234
Dynam ic View s concurrent view s/ users ...................................... 234
Dynam ic View s access via Webstart .............................................. 234
Dynam ic view s access control ........................................................ 235
Dynam ic view s via SSL .................................................................... 236
Container view s ................................................................................ 236
Container View custom ization best practices ............................... 237
Container View operations.............................................................. 238
Container View s access control ...................................................... 239
Container View exam ple ................................................................. 241
N od e View ......................................................................................... 241
Interface View ................................................................................... 242
N od e Status and Interface Status View s........................................ 242
Alarm View ....................................................................................... 243
N eighbor View .................................................................................. 243
VRRP View ........................................................................................ 243
OSPF View ......................................................................................... 244
Path View ........................................................................................... 244
H SRP View ........................................................................................ 245
Managing overlapping ad d ress spaces (OAD) ............................ 246
Delete an overlapping ad d ress spaces (OAD) .............................. 247
IPV6 .................................................................................................... 248
Port Ad min tool ................................................................................ 249
Aggregated port support ................................................................. 249
Increase JAVA heap size .................................................................. 250
Ad d , d elete, m anage or unm anage objects via URLs .................. 250
Troubleshooting d ynam ic view s .................................................... 251
xviii
Table of Contents
Dynam ic view s upd ate issues d ue to caching .............................. 251
Dynam ic view s registration files..................................................... 252
Dynam ic view s and ed ge connectivity .......................................... 252
Transfer MIB application build er apps to Dynam ic view s ......... 252
Custom ize d ynam ic view s-based alarm brow ser ......................... 253
22. Databases and Reporting ................ 254
Data w arehouse ................................................................................. 254
Solid d atabase .................................................................................... 255
Data w arehouse queries ................................................................... 255
Solid d atabase perform ance............................................................. 257
Determ ining the version of Solid .................................................... 257
Database com m and utilities ............................................................ 257
Reporter URLs ................................................................................... 258
Connecting Crystal Reports to N N M (Wind ow s) ........................ 258
Using Oracle for the d ata w arehouse ............................................. 258
ODBC Driver Manager Version errors........................................... 259
Accessing DW d ata d irectly from Microsoft Excel ....................... 260
Data w arehouse com patibility: N N M 6.0 to N N M 6.1 ................ 260
Disable the data w arehouse ............................................................. 260
Rebuild the d ata w arehouse ............................................................ 260
Rebuild the ET Database .................................................................. 261
Disabling N N M 6.2 d efault d ata collections ................................. 261
Extend and custom ize router perform ance reports ..................... 262
Extend ing the num ber of instances in the TopN report .............. 262
Support and contributed reporting tools ....................................... 262
Database m aintenance ...................................................................... 262
Delete old reports .............................................................................. 263
Rebuild DW after trim to recover space (UN IX) .......................... 263
Resolving errors w ith d atabase m aintenance program s ............. 264
Extrem e filtering in event d ata exports .......................................... 264
Database size lim itations .................................................................. 265
xix
Fognet’s Field Guide to OpenView NNM
Increase size of the binary event store (BES) ................................ 265
Clear the event d atabase (BES) ....................................................... 265
Dum p list of managed nod es into CSV file (UN IX only) ............ 265
Dum p a clean list of all m anaged nod es (UN IX only)................. 266
Dum p interface list for all nod es (UN IX only) ............................. 266
23. Distributed NNM ............................. 267
DIM limitations ................................................................................. 267
DIM overlap m od es .......................................................................... 268
Ports used by MS and CS:................................................................ 268
Polling through firew alls using DIM ............................................. 268
Filtering .............................................................................................. 268
ovfiltercheck, ovfiltertest and ovtopod um p ................................. 269
Exam ples of filter expressions ........................................................ 269
Using external files w ithin filters ................................................... 270
Filtering id iosyncracies .................................................................... 270
Exam ples of xnm topoconf com m and s .......................................... 271
Rem ove second ary objects from the d atabase .............................. 271
DIM set-up in a nutshell .................................................................. 271
Migrating to non-DIM w ith APA ................................................... 271
Using WEB interface w ith an MS as Managem ent Console ....... 272
CS view s from MS Alarm Brow ser ................................................ 272
24. High Availability and Backup ....... 273
Autom ated d atabase backups ......................................................... 273
ovbackup and OpenView d ata protector (Om niBack) ................ 274
ovresum e tim es out .......................................................................... 274
Running ovbackup.ovpl as non -root user (UN IX)....................... 274
ovbackup.ovpl and Data Warehouse interaction ......................... 274
ovdbcheck daem on not starting (MS SQL) ...................................... 275
Backing up SN MP configuration d ata ........................................... 275
xx
Table of Contents
Integrating N N M w ith OVO d atabase backup ............................. 275
Backing up the Solid d atabase ......................................................... 275
Map exports and im ports ................................................................. 276
Map snapshots ................................................................................... 276
ovtopod bsnapshot.ovpl .................................................................... 277
N ative backup and restore com m and s (UN IX) ............................ 277
DIM Configuration for a hot stand by system ............................... 277
N on-DIM configuration for a “hot stand by” ................................ 278
N N M as a highly available service ................................................. 280
Syslog in an H A environm ent ......................................................... 281
N N M and m ultiple N IC card s ........................................................ 281
Managing migratable IP ad d resses ................................................ 281
25. Firewalls and Security ..................... 283
N N M Ports used ................................................................................ 283
Reconfiguring ports used by N N M ................................................ 284
H ighly-secure netw ork m anagem ent scenario ............................. 284
N N M ICMP polls versus ping sw eep attacks ............................... 284
Security issue w ith ICMP pin g (UN IX only) ................................. 285
Discovering into subnets using NAT ............................................. 285
Monitoring DMZ d evices ................................................................. 285
Extend ed Topology configuration GUI passw ord ....................... 286
26. Product Details .................................. 287
N N M 8i ............................................................................................... 287
N N M 8i architecture ......................................................................... 288
Prod uct feature d eltas by N N M version ........................................ 290
Prod uct feature d eltas by OS ........................................................... 295
OS and softw are support m atrix ..................................................... 296
Stand ard ed ition vs. Ad vanced edition ......................................... 298
N od e d efinition ................................................................................. 298
N N M sm art plug-ins ........................................................................ 300
xxi
Fognet’s Field Guide to OpenView NNM
H P Softw are prod uct suite sum m ary ............................................ 300
Insight m anager integration ............................................................ 301
Core N N M prod uct num bers (U.S), versions 6.4-7.51................. 302
Support package num bers (N orth Am erica only) ....................... 304
Glossary.................................................... 305
Index ......................................................... 311
xxii
1. Introduction
This vade mecum is w ritten for field consu ltants, u sers and ad m inistrators
of the H P N etw ork N od e Man ager (N N M) softw are p rod u ct. The second
ed ition covers inform ation that is relevant for N N M p rod u ct versions 6.0
throu gh 7.53 w ith Interm ed iate Patch 20. It d oes not cover N N M 8i,
w hich is an entirely new and d ifferent p rod u ct (see p age 287). It w as
w ritten for those w ho seek a shortcu t to com m only u sed p rod u ct info
that is either m issing or obfu scated in the p rod u ct d ocs, and it covers
p ractical im p lem entation inform ation that can‟t be fou nd in any p rod u ct
d ocu m entation or the fine p rod u ct m anu al or reference p ages. This gu id e
w as gleaned from Op enView u sers and from the au thor‟s fifteen years of
com p iled notes on the p rod u ct.
So, w hat is N N M? N N M is a scalable enterp rise netw ork fau lt
m anagem ent tool that p rovid es SN MP m anager featu res, netw ork
d iscovery, OSI layer 2 and layer 3 top ology m ap p ing, netw ork d evice
statu s p olling, third p arty integration APIs and som e lim ited rep orting,
p erform ance and trend analysis cap abilities. It is the m ost p op u lar
enterp rise-class netw ork m anagem ent tool for m anaging IP netw orks.
The term s “Op enView ” and “N N M” have been u sed synonym ou sly for
m any years, bu t it has never been accu rate to m ix these term s. N N M is a
d iscreet p rod u ct w ithin the Op enView softw are p rod u ct su ite. A list of
som e of the m ore p op u lar and relevant H P Softw are p rod u cts are listed
on p age 300.
Shortly after its acqu isition of Mercu ry Interactive in 2007, H P d rop p ed
the u se of “Op enView ” in all its softw are p rod u cts and brand ed all
softw are u nd er the H P u m brella “H P Softw are.” “H P Op enView
N etw ork N od e Manager” thu s becam e “H P N etw ork N od e Manager.”
The au thor‟s exp ectation is that N N M w ill still be referred to as
“Op enView ” for at least another d ecad e.
The second ed ition contains exp ansions, corrections and coverage of new
featu res released in N N M 7.51 and 7.52 and 7.53 and interm ed iate
1
Fognet’s Field Guide to OpenView NNM
p atches. Alm ost all acronym s u sed w ithin this gu id e are exp and ed in an
attem p t at a com p rehensive glossary starting on p age 305. Feed back on
this gu id e is w elcom ed and encou raged and can be su bm itted to the
au thor throu gh the p u blisher.
Acknow ledgements
Rachel p roved ind isp ensable as chief ed itor for this book. The au thor‟s
sister Su e Wied orn d id the w ond erfu l cover art. The au thor‟s child ren,
Molly, Jake and Charlie su ffered greatly from a severe lack of attention
d u ring the w riting and u p d ating of this book. Sp ecial thanks for inp u t
nd
into the 2 ed ition go to Kevin May. Tu sen takk to N ils Johannessen
for p rovid ing cover to cover ed iting for the 2nd ed ition and
contribu ted som e great ad d itional m aterial. Kevin Sm ith of H P has
p rovid ed valu able insights for both ed itions. Finally, Tracy Avent of
H P p rovid ed both technical and m oral su p p ort and is ju st to o nice of a
fellow to skip a cred it herew ith.
Mu ch of this w ork w ou ld have been im p ossible w ithou t the
m entorship of som e of the eld er statesm en and statesw om en of the
Op enView Foru m listserv and greater Op enView com m u nity. In 2007,
OVForu m w as renam ed / rebrand ed to VivIt (vivit-w orld w id e.org)
About Fognet and the author
Mike joined Digital Equ ip m ent Corp in 1987 and in 1993 began w ork
on the DEC/ IBM OEM‟d p orts of N N M 3.31 to Alp ha OSF/ 1 and
N T/ Alp ha called PolyCenter and Tivoli N etView . Follow ing that,
Mike w orked for H P as a senior p ost-sales Op enView consu ltant. H e
fou nd ed Fognet in 1998 as a consu ltancy focu sed on enterp rise
m anagem ent. Fognet is an IT consu lting service com p any, d elivering
enterp rise
m anagem ent
p rod u ct
im p lem entations,
training,
architectu re and integration w ith a p rim ary focu s on the Op enView
N N M and Op erations p rod u cts. Visit Fognet‟s w eb site at
w w w .fognet.com .
Conventions used in this guide
Backgrou nd and foregrou nd p rocesses are listed italics. Com m and s are
listed in Courier font. File nam es are listed in regu lar font.
2
2. General Information
This section covers som e basic inform ation and p roced u res w hich are m ost
frequ ently u sed to ad m inister N N M.
Paths
The p aths to files u sed in this gu id e are show n u sing N N M stand ard
environm ent variables. The convention in this gu id e is to m ix UN IX
and Wind ow s-like p aths w ith the u nd erstand ing that the read er w ill
m ake the ap p rop riate translation for that read er‟s p latform . Thu s the
UN IX p ath for $OV_CON F/ C/ trap d .conf, w hen u sed in this gu id e is
the equ ivalent of %OV_CON F%\ C\ trap d .conf on the Wind ow s
p latform . For Wind ow s p latform u sers, the u se of environm ental
variables help s interp ret p aths w ithou t care as to the d irectory in
w hich N N M w as originally installed . Und er Wind ow s, all N N M
d irectories are relative to the installation d irectory. This is n ot the case
w ith the UN IX p aths, w here p er OSF conventions; files are laid ou t as
follow s:
/ etc/ opt/ OV/ share
/ var/ opt/ OV/ share
/ opt/ OV
Configuration files
Files that “grow”
Optional or program files
D efault passw ords
The follow ing p assw ord s are u sed to access variou s N N M su bsystem s:
SN MP read com m unity string:
Web GUI user:
Web GUI passw ord :
Data Warehouse DB user:
Data Warehouse DB password :
public
ovuser
ovuser
ovd b
ovd b
The Extend ed Top ology configu ration GUI p assw ord is in:
$OV_AS\ w ebapps\ topology\ WEB-IN F\ d ynam icView sUsers.xm l
3
Fognet’s Field Guide to OpenView NNM
OpenView environment variables
To activate the environm ent variables on Wind ow s so they can be u sed
to find d irectories from the com m and line, enter:
ov.envvars.bat
The absolu te p ath shou ld n‟t be necessary since the Op enView bin
d irectory is p laced in the p ath u p on p rod u ct install. On UN IX, u se the
ap p rop riate shell-sp ecific scrip t in the N N M bin d irectory. The
follow ing scrip ts can be called from the ap p rop riate com m and line
shell or from other scrip ts that them selves invoke these shells:
ov.envvars.sh
ov.envvars.csh
ov.envvars.pl
The com p lete list of environm ental variables can be fou nd w ithin the
files above. Selected environm ent variables for UN IX inclu d e:
OV_BACKGROUN DS
OV_BIN
OV_BITMAPS
OV_CON F
OV_CON TRIB
OV_DB
OV_DOC
OV_FIELDS
OV_H ELP
OV_JRE
OV_LIB
OV_LOG
OV_LRF
OV_MAN
OV_N EW_CON F
OV_REGISTRATION
OV_SH ARE_LOG
OV_SN MP_MIBS
OV_SYMBOLS
OV_WWW
OV_AN ALYSIS_CON F
OV_SH ARE_H TDOCS
OV_WWW_REG
OV_AS
4
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/
etc/ opt/ OV/ share/ backgrounds
opt/ OV/ bin
etc/ opt/ OV/ share/ bitmaps
etc/ opt/ OV/ share/ conf
opt/ OV/ contrib
var/ opt/ OV/ share/ databases
opt/ OV/ doc
etc/ opt/ OV/ share/ field s
var/ opt/ OV/ share/ help
opt/ OV/ jre/ jre1.4
opt/ OV/ lib
var/ opt/ OV/ share/ log
etc/ opt/ OV/ share/ lrf
opt/ OV/ man
opt/ OV/ new config
etc/ opt/ OV/ share/ registration
var/ opt/ OV/ share/ log
var/ opt/ OV/ share/ snm p_m ibs
etc/ opt/ OV/ share/ sym bols
opt/ OV/ ww w
etc/ opt/ OV/ share/ conf/ analysis
var/ opt/ OV/ w w w/ htd ocs
etc/ opt/ OV/ share/ w w w/ registration
opt/ OV/ tom cat/ jakarta-tom cat-4.0.4
General Information
Extending OV environment variables
There are no restrictions to ad d ing cu stom variables to the shellsp ecific scrip t in $OV_BIN/ov.envvars.* on either the UN IX or
Wind ow s server nod es.
N ote that Wind ow s environm ental variables are not case sensitive
w hereas UN IX environm ental variables are.
N N M environment variables usage restrictions
Environm ent variables cannot be u sed in action callbacks. Environm ent
variables can be u sed in tru sted Cm d s.conf files.
Customize daemons via local registration files (LRF)
Local registration files are u sed to set op tions and behaviors for
Op enView ‟s backgrou nd d aem ons. LRFs are read by the com m and s
ovaddobj and ovdelobj to register or d eregister an N N M m anaged
p rocess w ith ovspmd.
LRF changes that corresp ond to d aem on com m and line op tions are
fou nd betw een the second and third set of colons in the file. So in the
follow ing ovw d b.lrf file line, the com m and line argu m ent is “ -O”.
OVs_YES_START::-O:OVs_WELL_BEHAVED:15:PAUSE
When u p d ating an LRF, the general p roced u re to m ake the change
effective is:



Backup the original LRF
Ed it the LRF and enter changes
Run:
ovaddobj $OV_LRF/<daemon>.lrf
ovstop <daemon>
ovstart <daemon>
A com m on error is that either the ovaddobj com m and is not ru n from
the $OV_LRF d irectory, or the fu ll p ath to the LRF file is not sp ecified .
Som e Op enView d aem ons have d ep end encies on each other, so w hen
an ovstop is issu ed for a p articu lar d aem on, several other d aem ons
m ay also be shu t d ow n.
Dep end ent d aem ons are listed in the second colon -sep arated field in
the LRF file. In general, it is a good p ractice to stop all u ser GUI
sessions w hen issu ing ovstop com m and s. To m ake su re that all
5
Fognet’s Field Guide to OpenView NNM
ap p rop riate d aem ons are ru nning after starting or starting any
d aem ons, ru n ovstatus –c.
Debu g any p rocess configu red in a LRF by u sing the -debug x op tion
in the forth LRF field . “x” as a valu e betw een 1 and 5 w ith increasing
levels of verbosity.
A history of ovaddobj and ovdelobj op erations is held in the
$OV_CON F/ ovsu f file. In som e cases, this file can grow large, as in
w hen ovaddobj and ovdelobj op erations are u sed in u ser-cu stom ized
sched u led op erations to control N N M behaviors like d iscovery, etc. In
this case, the ovsu f shou ld be cleared ou t on occasion to im p rove
p erform ance. When d oing this, p reserve the one line low est in the file
for each d aem on listed .
Application registration files (ARF)
Any file p laced in $OV_REGISTRATION / C is read by the ovw p rocess
to register ovw GUI m enu -bar item s. The syntax is d escribed in d etail
in the OVw RegIntro m an/ ref p ages. Use these sim p le ru les to
gu arantee su re sm ooth sailing:
1.
2.
3.
4.
Make backup copies of registration files to be m od ified , but do not
store the backups in the registration d irectory tree as they w ill be read
and cause errors
Any filenam e supported by the OS is read by ovw
Test syntax w hen m aking changes using:
regverify –arf <arf filename>
See page 206 for proced ure to create alternate ARF trees
The Menu Bar item attribu tes in ARF files inclu d e: Selection
cap abilities based on object DB field s, H otkeys, Preced ence nu m ber,
and Selection ru le sp ecifications. The MIB Ap p lication Bu ild er is frontend ARF file generator d esigned to w ork w ith SN MP MIB valu es. The
follow ing d irectory is reserved for MIB ap p lication bu ild er ap p s:
$OV_REGISTRATION / C/ ovm ib
For som e excellent exam p les, see the N N M m anu al:
Creating_and_Using_Reg_Files.pd f
6
General Information
Symbols, OID types and fields (FRF)
oid _to_sym p rovid es the m ap p ing of d evice typ es to the sym bol u sed
to rep resent those d evices. oid _to_typ e and H Poid 2typ e p rovid e the
m ap p ing of d evice typ es to how to treat the d evices in the IPMAP
top ology. In version 7.0 and greater, the oid _to_sym _reg d atabase w as
introd u ced to facilitate better integration w ith N N M ET. For backw ard
com p atibility, the oid _to_sym configu ration file is still read first w hen
IPMAP starts.
Field registration files (FRFs) d efine all the field s in the object (ovwdb)
d atabase. FRF u p d ates are necessary w hen a new sym bol m ap p ing is
d esired or to ad d u n iqu ely id entifying attribu tes to the object d atabase.
The oid _to_typ e and H Poid 2typ e m ap p ings u se tw o object d atabase
field valu es: “vend or”, and “SN MP agent”, to assign top ology ru les
for d evices w hich m ap to those field s. For alm ost all N N M
d ep loym ents, there w ou ld p robably be agents and vend ors that are not
listed in oid _to_typ e, so it m ay be d esirable to ad d these m issing
valu es via the FRF p rocess.
If there is a need sim p ly to m ake su re d evices for an u nd efined SN MP
agent are m ap p ed p rop erly in IPMAP, it is not requ ired that FRFs be
u p d ated . A shortcu t is to assign p re-existing Vend or and SN MP agent
valu es to the new ly ad d ed SN MP agent OID in the oid _to_typ e files
that is alread y associated w ith a sym bol that satisfactorily rep resents
the object. The only p roblem w ith this shortcu t is that object and
top ology d atabase searches and rep orts m ight p rod u ce incorrect
resu lts.
When ad d ing a new entry, d eterm ine if the vend or and / or agent is
alread y in the object d atabase (a p re-requ isite) by p erform ing find
op erations in the ovw GUI. Alternatively on UN IX, issu e follow ing tw o
com m and s:
ovobjprint -e `ovobjprint -f |grep vendor |cut -f1`
ovobjprint -e `ovobjprint -f |grep SNMPAg |cut -f1`
Su m m ary of oid _to_typ e and H Poid 2typ e top olog y attribu tes:
B
D
S
N
d
Brid ge
Cisco H SRP (ip ad d r‟s not in ip.ipAd d rTable)
2ndary ad d rs. Do not d elete 2ndary If‟s
Ignore unnum bered interfaces on these d evices
Dorm ant oper status - Cisco ISDN
7
Fognet’s Field Guide to OpenView NNM
G
H
I
L
M
T
U
Gatew ay
H ub
Ignore SN MP
Allow Loopback con nections
N ot a Gateway
Term inal Server
Discover as unmanaged
The follow ing global d efau lts for oid _to_typ e and H Poid 2typ e are
u sed to sp ecify global top ology treatm ent for classes of d evices that
d on‟t otherw ise m ap to sp ecific OID entries in the file:
DEFAULT_SN MP -- Supports SN MP but d oes not have a more specificOID entry
DEFAULT_IP -- Supports IP and otherw ise w ould not have a matching entry
DEFAULT_IPX -- Supports IPX and otherw ise would not have a m atching entry
DEFAULT -- Any nod e w hich otherw ise w ould not have a matching entry
For exam p le, to instru ct N N M to d iscover SN MP-su p p orted nod es
that are not listed in the files as u nm anaged , u se:
DEFAULT_SN MP:::U
To u nm anage Wind ow s servers and / or w orkstations w ith SN MP
tu rned on, change these entries:
1.3.6.1.4.1.311.1.1.3.1:Windows NT:U # Pre NT SP2
1.3.6.1.4.1.311.1.1.3.1.1:Workstation:Windows NT:U
1.3.6.1.4.1.311.1.1.3.1.2:Server:Windows NT:U
1.3.6.1.4.1.311.1.1.3.1.3:PDC:Windows NT:U
To ad d a “vend or” FRF:
Ed it the file $OV_FIELDS/ C/ ovw _field s and ad d the new
vend or, an exam p le of the section follow s :
Field "vendor" {
Type
Enumeration;
Flags
capability, general, locate;
Enumeration "Unset",
"Hewlett-Packard",
"3Com",
"ACC",
"Allied Telesyn"
To ad d an “SN MP agent” FRF:
Ed it the file $OV_FIELDS/ C/ snm p _field s and ad d the new
agent, an exam p le of the section follow s:
8
General Information
Field "SNMPAgent" {
Type Enumeration;
Flags
capability, general, locate;
Enumeration "Unset",
"HP 3000/XL",
"HP 386",
"HP 700/[R]X X-Terminal",
"HP 9000/HP-UX",
"HP Bridge",
To m ake su re no syntax errors have been introd u ced , ru n the follow ing
com m and and look for errors:
Wind ow s:
UN IX:
regverify
ovw –verify >/dev/null
Once syntax has been verified , u p d ate the object d atabase u sing:
ovw -fields
Once valid Agent and Vend or field s are registered , the new field
entries can be u sed in the oid _to_typ e file and new ly-d iscovered
d evices shou ld be m ap p ed accord ing to the ru les.
To u p d ate alread y d iscovered objects w ithou t having to red iscover
them , exit ovw and ru n:
ovstop netmon
ovtopofix -u -o <sysObjectID>
ovstart netmon
To ad d cu stom ized sym bols, see the p roced u re “Create a Sym bol
Registration File” in the Ap p end ix of th e N N M Gu id e: Managing You r
N etw ork.
Application defaults update procedure (app-defaults)
UN IX:
Ed it the ap p rop riate file in the $APP_DEFS d irectory.
Wind ow s:
Ed it the ap p rop riate registry key u nd er:
H KEY_LOCAL_MACH IN E if the value is global for all users, or :
H KEY_CURREN T_USER, if d ifferent users need to be able to use d ifferent application
d efault values.
The assorted classes are fou nd u nd er:
H KLM\ SOFTWARE\ or H KCU\ SOFTWARE\
9
Fognet’s Field Guide to OpenView NNM
\ Hew lett-Packard \ OpenView \ Netw ork N od e Manager
The ap p -d efau lts files (UN IX) or classes (Wind ow s) are:
OVw
XN m
XN mappmon
XN m events
XN m graph
XN mtrap
ovw GUI settings, e.g. Map colors, fonts
GUI settings for MIB brow ser, collector, etc
GUI settings for tables and form s
GUI settings for alarm brow ser
GUI settings for grapher
GUI settings for event configuration
Troubleshooting Window s paths
In N N M event configu ration action callbacks, backslashes m u st be
escap ed and sp aces m u st be qu oted , for exam p le:
C:\\‖Program Files‖\\‖HP OpenView‖\\program.bat
In N N M filter files, the colon m u st be escap ed and the backslashes
rep laced w ith regu lar slashes.
servers "servers" {d\:/ov/conf/coreservers.txt}
In N N M ARF files, qu otes are need ed for sp aces and m u st be escap ed ,
bu t all the follow ing exam p les w ill w ork:
Application "Wordpad" { Command "C:/\"Program
Files\"/Windows\" \"NT/Accesories/wordpad";}
Application "wordpad" { Command "\"C:/Program Files/Windows
NT/Accesories/wordpad\""; }
Application "webpage" { Command "C:\\PROGRA~1\\
INTERN~1\\iexplore.exe http://webpage.html" }
Finally, if none of the above seem s to w ork, ad d the d irectory location
of the execu table to the system p ath, or u se the 'Ap p Path ' registry key,
then sim p ly call the execu table nam e w ithou t sp ecifying the p ath.
Force or restart an ET discovery
Before attem p ting this op eration, check to see if a d iscovery is
cu rrently ru nning: from H om e Base, select the Discover y Statu s tab.
Check d iscovery statu s from the com m and line by ru nning:
ovstatus –v ovet_disco
Restart or force ET d iscovery only w hen u nd erlying configu ration
changes have been m ad e that m ay requ ire su ch an action. Make su re
that any d evices that m ay be the target of a force ET d iscovery are
p rop erly d iscovered by netmon first.
10
General Information
ET d iscovery can be initiated from H om ebase: select Discovery Statu s,
then ET Configu ration, then Initiate Fu ll Discovery N ow . This can also
be accessed from ovw via: Op tions->Extend ed Top ology Configu ration.
Or, the w eb URL is:
http://<nnm server>:7510/topology/etconfig
To force ET Discovery from the com m and line, ru n:
etrestart.ovpl
or:
etrestart.ovpl -verbose –disco
Be p rep ared to w ait a w hile for d iscovery to com p lete.
The p rogress of d iscovery can be follow ed on the Discovery Statu s
p age. If changes have been m ad e that requ ire an ET red iscovery, and a
d iscovery is alread y in p rogress, u se:
etrestart.ovpl -force
To refresh H om ebase and APA statu s view s, ru n:
ovstop ovas
ovstop ovet_poll
ovstart ovas
ovstart ovet_poll
By d efau lt, ET w aits u ntil netmon has fou nd 2500 changes in a netw ork
before ET ru n s a new d iscovery. To change this so it ru ns at a
sched u led tim e, go the ET configu ration (see above) and tu rn off
“Enable d iscovery for a sp ecified nu m ber of N N M changes.”
Then tu rn on “Enable recu rring d iscovery.” Sched u le a d iscovery
d aily, and click Ap p ly. N ote that low ering the threshold m ay have a
p erform ance cost.
If ET red iscovery isn‟t p rop erly d iscovering p articu lar d evices, see the
section on im p roving ET d iscovery on p age 221.
11
Fognet’s Field Guide to OpenView NNM
Re-do discovery (start from scratch)
Re-d o d iscovery after m aking m ajor changes su ch as sw itching from
files-based looku p s to DN S, or to recover from su sp ected corru p tion of
the object, m ap , top ology, or ET d atabases.
Before p erform ing this op eration, consid er exp orting m ap s to save
cu stom izations (see p age 276).
Review any ad d itional changes that m ay p rofou nd ly affect top ology,
su ch as reclassifying the treatm ent of a m ajor class of connector
d evices by OID, DN S changes, etc.
To ju st d elete the d atabases, ru n the follow ing com m and :
$OV_CONTRIB/deleteOVDB/deleteOVDB.ovpl
To m anu ally start from scratch:
1.
2.
3.
4.
5.
6.
7.
8.
9.
Im portant: close all OpenView GUIs
ovstop –c
Remove the databases & log files:
UNIX: cd $OV_DB; rm –rf openview; rm –rf nnmet
Wind ow s: d elete all the d irectories below :
\ databases\ openview
\ databases\ nnm et
If exporting topology, run :
ovtoposql -R i|o -C
Clear the SN MP cache:
xnmsnmpconf -clearCache
Restart all the OV daem ons: ovstart -c
Start the user interface:
ovw &
Re-initiate ET:
setupExtTopo.ovpl
Logging N N M errors to Window s event log
The ovtracelog u tility is p rovid ed for logging N N M excep tions throu gh
the Wind ow s tracing and logging facility. See the ovtracelog reference
or m an p age for d etails.
Running ovstart and ovpause as non-root (UN IX)
Use the follow ing step s to allow non -root u sers to issu e the ovstart
and ovstop as w ell as ovpause or ovresume com m and s.
1.
12
Make sure root is the owner of ovstart and ovstop
General Information
2.
3.
4.
5.
6.
7.
8.
9.
chmod 4555 ovstart ; chmod 4555 ovstop
Create a file called $OV_CON F/ ovstart.allow
chmod 400 ovstart.allow ; chown root ovstart.allow
In the file put a user nam e or UN IX user ID
chown root:sys ovstart ovstop ovstatus
chown root:sys ovpause ovresume
chmod 666 $OV_LOG/ovstart.log
Consult the $OV_LOG/ ovstart.log for possible errors.
Common log files
The follow ing are com m on log files is $OV_LOG:
Setup.log
ovactiond .log
ovalarm srv.trace
ovcapsd .log
ovrepld .log
ovbackup.log
Pm d .log0
snm pCol.trace
ovdw *.log
Troubleshoot installation (Window s)
Troubleshoot automatic actions
Troubleshoot xnm events
Troubleshoot DMI/ RDMI (Window s)
Troubleshoot MS-CS upd ates
Troubleshoot ovbackup.ovpl actions
Troubleshoot postmaster daemon
Troubleshoot raw data collection
Troubleshoot data warehouse and reporting
The follow ing are com m on log files is $OV_PRIV_LOG:
httpd *.log
ovas_err.log
ovrequestd.log
Troubleshoot w eb server
Troubleshoot Dynam ic View s
Troubleshoot Data Warehouse feed s
N N M product documentation
User m anu als are fou nd in:
$OV_WWW/ htdocs/ C/ Manuals
Online help :
Unix:
http:/ / <nnm server>:3443/ OvCgi/ OvWebH elp.exe
Wind ow s:
http:/ / <nnm server>/ OvCgi/ OvWebH elp.exe
Und er Wind ow s, the online help facility can be invoked d irectly even
if N N M is not ru nning:
13
Fognet’s Field Guide to OpenView NNM
%OV_H ELP\ C\ NN M.hlp
Dow nload u ser m anu als:
http:/ / ovw eb.external.hp.com/ lpe/ doc_serv
Ad d itional ECS Manu als:
Selected RFC‟s:
White Pap ers:
Release N otes:
Release N ote Up d ates (V7.5):
$OV_DOC/ ECS/ C
$OV_DOC/ RFC
$OV_DOC/ WhitePap ers
$OV_WWW / htd ocs/ C/ ReleaseN otes/
http:/ / ovw eb.external.hp.com/ nnm / N N M7.5/ relNoteUpd/ relNoteUpd ate.htm
Man Pages (Linux)
Man p ages for d aem ons and certain com m and s are p laced u nd er
/ op t/ OV/ m an/ m an1m for N N M and / op t/ OV/ m an/ m an1m .Z for
ECS tools. The 1m and 1m .z m anu al sections m ay not be d efined by
d efau lt u nd er Linu x varients. Sp ecify the section in the com m and line
to get arou nd this issu e, for exam p le:
man 1m netmon
man 1m.Z ecsmgr
Alternatively, 1m and 1m .Z m ay be ad d ed to the MAN SECT variable
in the / etc/ m an.config file, for exam p le:
MAN SECT 1:8:2:3:4:5:6:7:9:tcl:n:l:p:o:1m :1m .Z
N N M patch information
Em ail notifications for p atch u p d ates:
http:/ / support.openview .hp.com/ em ailReg.jsp
Determ ining w hat p atches are installed :
Wind ow s:
UN IX:
14
Control Panel-Ad d / Rem ove Program s
sw list -l fileset | grep "^ # PH "
3. Common Issues
This section covers som e of the m ore com m on issu es encou ntered d u ring
initial installation and setu p -u p of N N M.
Troubleshooting resources
The follow ing tool can be u sed on any N N M p latform to check p atch
levels. Alw ays try to keep N N M u p -to-d ate on p atch levels:
$OV_SUPPORT/checkPatch.ovpl
The m ost com p rehensive d atabase of solu tions to com m on N N M
p roblem s is located at the H P IT Resou rce Center at:
w w w .itrc.hp.com
Using the ITRC-em bed d ed search engine, how ever, is often slow and
yield s p oor resu lts. Fortu nately, the entire ITRC d atabase is m ined by
Google, so w hen an error occu rs that is very generic, ad d ing “ITRC” to
the search term in Google can often help narrow d ow n the issu e.
Another excellent resou rce is the VivIt (form erly OVForu m ) listserv at:
w w w .vivit-w orld w id e.org (form erly w w w .ovforum .org )
The foru m p rovid es a few sp ecial interest grou p s, bu t by far, the m ost
active is the general foru m w hich covers all Op enView p rod u cts. In
p ractice, m ost of the traffic p ertains to N N M. The ovforu m listserv is
m aintained by the Op enView Foru m International u sers grou p , w hich
is an ind ep end ent organization that has no d irect ties to H P. For this
reason, it is a good p lace to su bm it qu estions that are less biased by
H P. For exam p le, qu estions p ertaining to com p araisons of Op enView
w ith com p eting p rod u cts or d iscu ssions abou t the qu ality H P p rod u cts
and services. Also, u ser exp eriences and cu stom izations are m ore
likely to be fou nd on the listserv. Conversely, very few H P em p loyees
p articip ate in the ovforu m listserv . Qu estions p osed to the ITRC,
15
Fognet’s Field Guide to OpenView NNM
how ever, are very likely to be answ ered d irectly by H P su bject m atter
exp erts.
Licensing issues due to upgrades, migrations, homing
License u p d ates are requ ired
circu m stances:
from
H P u nd er
the follow ing
 When upgrad ing from 6.x to 7.x
 When changing IP ad d ress
 When changing server OS
If an N N M server has m ore than one IP ad d ress, the p referred IP
ad d ress can be sp ecified u sing the N N M_IN TERFACE keyw ord in the
follow ing file:
$OV_CON F/ ov.conf
Changing N N M server hostname or IP address
Generally, changing the N N M server nam e or IP is som ething to
avoid , m ostly becau se N N M registers itself (if it can) as a trap recip ient
on SN MP agents it d iscovers, bu t also for all the reasons listed in the
section on N N M and N am e Services (see p age 24). Changing IP
ad d ress m eans that trap s m ay be lost, p articu larly if trap d estinations
have been set m anu ally for m any d evices. Be aw are that on m any
d evices, if ad d ing a trap recip ient by FQDN , this is converted to the IP
ad d ress in the d evices configu ration. An IP ad d ress change also
requ ires that the p rod u ct license be u p d ated . To change the hostnam e
and/ or IP ad d ress:
1.
2.
3.
4.
5.
6.
7.
8.
Stop NN M processes using ovstop
Change the nam e in: $OV_DB/ openview / ovw d b/ ovserver
Change the nam e in all the $OV_CON F/ *auth* files
Change the server name in the OS
Run: xnmsnmpconf -clearC
Update license if IP Ad d ress changed
Update agent trap d estinations if IP ad d ress changed
Start processes using ovstart
Installation issue: virtual directories not created
Sym p tom s are that “the p age cannot be fou nd ” or “Und er
constru ction” ap p ears w hen the w eb GUIs first lau nched . This
hap p ens on Wind ow s installations w hen N N M 7+ is installed into an
environm ent w here anti-viru s softw are p revents VBscrip ts from
ru nning.
16
Common Issues
To resolve this issu e, d isable any p rogram s that block vbscrip ts then
locate the setupIIS.vbs scrip t from the new config su bd irectories on
the installation m ed ia, then ru n:
cscript setupIIS.vbs "C:\Program Files\HP OpenView\NNM" –
If issu es p ersist, installation registry file entries m ay need to be
u p d ated . There are articles w hich d escribe the u p d ate p roced u re on
ITRC.
Installation issue: ov as not running
Typ ically, the root cau se is w ith the w eb server‟s configu ration. The re
are m any p ossible things that can go w rong. In trou bleshooting,
alw ays look to the w eb server‟s health first. Is it ru nning? Does the
event log (or syslog) contain p ertinent inform ation? Do the w eb server
logs look O.K.?
Check the ovas.log file. ovas is the d aem on that serves d ynam ic view s.
If ovas.log show s an error like this:
ld.so.1: ovas: fatal: relocation error: file
/opt/OV/jre/jre1.4/lib/sparc/libjava.so: symbol
VerifyFixClassname: referenced symbol not found
The p roblem is that LD_LIBRARY_PATH is not set correctly, p ossibly
set by another ap p lication (e.g. Checkp oint). Reset it to:
/ opt/ OV/ lib
Another com m on p roblem cau sing ovas to fail to start is that another
ap p lication (p robably one that u ses Tom cat) has grabbed the Tom cat
server p ort. In N N M V7.01+ this is p ort 8005. To resolve, change the
Server Port nu m ber from 8005 to som e other u nu sed p ort nu m ber,
then restart the ovas d aem on. The Server Port is sp ecified in the
follow ing file:
$OV_AS/ conf/ server.xm l
Installation issue (UN IX): Using N FS mounted CD s
When exp orting a rem ote CD d rive to be u sed for installing N N M,
'root' access to the d rive m u st be granted . Use a line sim ilar to the
follow ing on the m achine that has the CD d rive installed on it:

HP-UX in / etc/ exports:
<m ount_point_of_CD> -ro,root=<m achine_nam e_1[:m achine2]>
17
Fognet’s Field Guide to OpenView NNM


Solaris in / etc/ d fs/ d fstab
share -F nfs -o root=<m achine_nam e> <m ount_point_of_CD>
Red Hat Linux A dvanced Server 2.1 in / etc/ exports
<m ount_point_of_CDROM> <m achine_nam e> (ro,all_squash)
Installation issue (Window s): N on-standard directory
N N M inserts inform ation abou t its install location into the registry. If
N N M is be installed in a non-stand ard location (other than
%System Drive%/ Program Files/ H P Op enView ), that su bd irectory
shou ld be created p rior to installation. This w ill allow the nonstand ard location to be u sed to store d ata (rather than in
%System Drive%/ Program Files/ H P Op enView / d ata). Create the
<install location>/ d ata d irectory before installation.
Installation issue (Window s): Terminal Services
When installing N etw ork N od e Manager 7.5x u nd er Term inal Services,
p roblem s m ight exist w here %System Root% d oes not correctly resolve
in Com m and Prom p t w ind ow s. If this occu rs, a reboot is necessary.
Unable to dow nload JRE through firew all
Recent versions of N N M (7.01+) d o not com e bu nd led w ith JRE and
au tom atically d etect if a com p atible version is installed on the server. If
not, the install p rocess m ay attem p t to access su n.com to d ow nload the
bits. Som e firew all ru les requ ire a resolvable DN S nam e for the
d ow nload , bu t the Su n d ow nload site for JRE retu rns a virtu al host.
Resolvable locations for JRE are available and can be fou nd in a w eb
search. One su ch site is:
http:/ / download .au.kd e.org/ pub/ java-sun
If firew all ru les w on‟t even p erm it that, try to d ow nload and ru n the
“Wind ow s Offline Installation” of the ru ntim e environm ent from :
http:/ / java.sun.com / j2se/ 1.4.2/ dow nload.htm l
D evices discovered in unmanaged state
Objects that are beige in the m ap s are u nm anaged objects. Whenever a
nod e or interface is d iscovered w hose IP ad d ress is in a su bnet that
w as p reviou sly u nd iscovered by netmon, a new netw ork sym bol is
p laced on the internet m ap in the u nm anaged state. This is to p reven t
ru naw ay d iscovery. N ew netw orks need to be selected and then
m anaged from w ithin the ovw m ap .
18
Common Issues
When rand om d evices are d iscovered as u nm anaged , it cou ld be that
the server‟s license lim it has been reached . Ru n the follow ing
com m and to d eterm ine the nu m ber of d iscovered objects and licensed
object cou nt: ovtopodump –l. Finally, d evices m ay be p laced on the
m ap in the u nm anaged state d u e to the “-U” configu ration flag in the
oid _to_typ e or H Poid 2typ e files. See p age 7 for m ore info on these
files. Use a netmon seed file as a w ay to allow for the d iscovery of
selected netw orks as m anaged . Also, scrip ts that search for
u nm anaged d evices an d then m anage them externally can be w ritten
to sim u late the N N M version 3.31 behavior of d iscovering the e ntire
netw ork by d efau lt.
See p age 41 for inform ation abou t m anaging objects externally and
p age 42 for m ore abou t seed ing netmon.
Cleaning orphaned objects from object database
N N M d atabase irregu larities m ay m anifest them selves in several w ays.
For exam p le, an error m essage ind icating that a p articu lar object
cannot be ad d ed to the top ology u sing loadhosts is one ind ication the
object exists in one or m ore N N M d atabases bu t has som ething w rong
w ith it or is ou t of sync w ith another N N M d atabases.
These sorts of issu es are also com m on in environm ents w here DN S is
very d ynam ic or som etim es m isconfigu red . It is a good general
p ractice to rep eat this p roced u re occasionally:
Open all existing m aps and allow them to synchronize, close.
ovstop netmon
xnmsnmpconf –clearC
ovw –mapcount –ruDv
ovtopofix –a
ovstart netmon
Open all existing m aps and allow them to synchronize.
D aemons ovet_da* not running (ovstatus)
After enabling Extend ed Top ology, new d aem ons are registered
throu gh the ARF object m anager. The ovstatus com m and m ay retu rn
“N ot Ru nning” for d aem ons w ith nam es like ovet_d aDetails,
ovet_d aCDP, ovet_d isco, etc. These d aem ons are su p p osed to be not
ru nning m ost of the tim e, and m ight only d isp lay as ru nning w hen
they are called to ru n d u ring an ET d iscovery cycle.
19
Fognet’s Field Guide to OpenView NNM
They m ay also show as ru nning soon after an ovstart com m and is
issu ed , bu t they m ay su bsequ ently show as not ru nning w hen and
ovstatus com m and is issu ed .
N etscape and N N M co-existence (UN IX)
N etscap e m ight allocate the entire color m ap w hen started , w hich m ay
w hack ou t ovw colored icons and cau se the error seen im m ed iately
below . To force N etscap e to u se a p rivate color m ap , start it u p u sing:
netscape –install
Alternatively, the m axim u m nu m ber of colors, and/ or the u se of a
p rivate color m ap can be sp ecified in the X Ap p Defau lts resou rce files
for N etscap e.
Cannot allocate 128 colors - using monochrome images
On UN IX p latform s, a m essage sim ilar to above m ay occu r w hen there
are not enou gh colors allocated to the colorm ap that N N M is
requ esting to p rop erly d isp lay icons, etc. A sim ilar m essage m ay be:
Allocation errors: 32 colors (128 requested ).
To resolve, from w ithin CDE, bring u p the Style Manager. Und er
Color, there is an op tion for “N u m ber of Colors.” There shou ld be an
op tion u nd er that to set “More colors for ap p lications.” Also, see the
section im m ed iately above and u se the “-install” N etscap e op tion, if
u sing netscap e. If ru nning X over a virtu al w ind ow ing server su ch as
Exceed or Reflections, be su re the server ap p lication is configu red to
u se a color d ep th of 24.
Unable to load any useable font set (UN IX)
Sym p tom : When lau nching an xnm grap h w ind ow :
Warning: Cannot convert string "-dt-interface system-medium-rnormal-m*-*-*-*-*-" to type FontSet
Warning: Unable to load any useable fontset
To resolve, ru n:
xrdb -merge /usr/lib/X11/app-defaults/Xnmgraph
20
Common Issues
xterm: unable to locate a suitable font (UN IX)
Create a file in the hom e d irectory of the u ser called .Xd efau lts and
inclu d e the follow ing line in the file:
xterm*Font:-*-lucidatypewriter-bold-r-*-*-12-*-*-*-*-*-*-*
Missing charsets in string to fontset (Solaris)
Sym p tom : On lau nch of ovw, the follow ing errors retu rn:
Warning: Cannot convert string "-dt-interface system-medium-rnormal-s*-*-*-*-*-" to type FontSet
Warning: Missing charsets in String to FontSet conversion
Warning: Unable to load any usable fontset…
Resolu tion:
1. Startu p the font server on the Solaris w orkstation:
/ usr/ openw in/ bin/ fslsfonts -server <N N M Server>:7000
2. Enable the font server on the Solaris w orkstation:
/ usr/ openw in/ bin/ xset+fp tcp/ <N N M Server>:7000/ all
Missing charsets in string to fontset (HP-UX)
Sym p tom : On lau nch of ovw, the follow ing errors retu rn:
Warning: Cannot convert string "-dt-interface system-medium-rnormal-s*-*-*-*-*-" to type FontSet
Warning: Missing charsets in String to FontSet conversion
Warning: Unable to load any usable fontset…
Resolu tion: To tu rn on the H P-UX font server:
1.
2.
3.
4.
5.
6.
7.
On the H P-UX server, ed it: / etc/ X11/ fs/ config
Append the follow ing to the end of the "catalog=...." line:
/usr/dt/config/xfonts/C
Save the file and ed it: / etc/ rc.config.d / xfs
Set the follow ing variable in the above file:
RUN_X_FONT_SERVER=1
Save the file and run the follow ing comm and s:
/sbin/init.d/xfs stop
/sbin/init.d/xfs start
On the target system enter:
xset +fp tcp/<Server IP Addr>:7000 1>/dev/null
xset fp rehash
If xset is not in the path, append $XDIR environm ent variable.
N TLM - passw ord needed - firew all: unknow n
Often seen as an installation issue, this is a known issue with JAVA
1.4.2_01. To resolve, explicitly set the proxy server by selecting "Bypass
21
Fognet’s Field Guide to OpenView NNM
proxy server for local addresses" dialog under LAN settings under the
connections tabs in IE Internet Options when using a script for proxy
detection. Similarly, if the local user account specified for anonymous
IIS access is disabled due to site security policies, this needs to be
changed.
Troubleshooting daemons using ovstart and ovstatus
To get m ore statu s for a d aem on: ovstatus –v <daemon>
To get highest verbosity (UN IX only): ovstart –v –V –d <daemon>
Automatically detect and restart OV daemons that die
The follow ing log-only Op enView enterp rise event can be u sed to
d etect and log failing d aem ons that are registered via ARF and “WellBehaved ”:
OV_AppUngracefulExit .1.3.6.1.4.1.11.2.17.1.0.59179058
The fou rth varbind contains the nam e of the d aem on that exited
u ngracefu lly, so in ad d ition to logging this event, set u p an au tom atic
action that execu tes: ovstart $4
Filtering on custom ov w db fields
The N N M Gu id e to Scalability and Distribu tion Ap p end ix A Table A -2
lists objects and attribu tes that can be u sed in a filter. The
IPMAP_FILTER_FROM_OVW environm ent variable allow s filtering on
cu stom ized field valu es and those ad d ed by third p arty p rod u cts. See
p age 3 for inform ation on setting environm ent variables.
Working w ith multiple JAVA versions
Mu ltip le versions of JAVA m ay be installed on the N N M server or on
any clients that are u sed to access the N N M server. See the su p p ort
m atrix on p age 296 to d eterm ine w hich version of JAVA is requ ired for
a p articu lar version of N N M. On the client sid e, the environm ent that
invokes N N M need s to be configu red to p oint to the ap p rop riate
version of JAVA.
On UN IX system s set the environm ent variable $JAVA_H OME to
p oint to the binary d irectory for the su p p orted JAVA installation, for
exam p le: / op t/ java1.4/ bin. On Wind ow s system s, set the com p atible
version via: Control Panel->Java Plu g-in. Often, other JAVA
ap p lications requ ire d ifferent versions than those that are com p atible
w ith N N M. Som e u sers resort to configu ring m u ltip le vend ors‟
22
Common Issues
brow sers w ith sep arately-configu red versions of JAVA. While
certainly inconvenient, it solves som e cross-ap p lication issu es. It is
ad visable w hen installing m u ltip le JAVA versions to install them
increm entally from low est to highest version.
N ote that Microsoft Internet Exp lorer brow sers su p p ort only a single
installation onto any one system . Many other brow sers, how ever, can
be installed m u ltip le tim es onto the sam e system , so each installed
instance can be configu red to p oint to sep arate versio ns of JAVA.
Client-sid e su p p ort for m u ltip le JAVA versions is im p roved w ith
Version 1.5 and greater. The p rocess is to cop y the d irectories for the
p re 1.5 versions from w here they w ere installed to a new location.
Deinstall all p re 1.5 versions (Control Panel, Ad d / rem ove p rogram s)
then install a 1.5+ version, w hich can be fou nd at:
w w w .java.com / en/ d ow nload / w ind ow s_ie.jsp .
Cop y/ m ove the p reviou s version d irectories to a su bfold er of the
new ly installed 1.5+ version. Start Control Panel/ Java. Select
Java/ View u nd er Java Ap p lication Ru ntim e Setings. Choose Find , and
search and find the javaw.exe for the p reviou s version(s), Verify that
the ap p lication d etects correct info u nd er Platform and Prod u ct. Select
OK to exit Java Control Panel.
23
4. NNM and Name Services
N N M relies heavily on looku p s for m ap p ing nam es to ad d resses and
vice versa. The section on netmon d iscovery on p age 36 exp lains the
im p ortance of stable and accessible nam e services to N N M.
Using / etc/ hosts or LMhosts files based looku p s w orks w ell, bu t
p rovid es very lim ited ability to u se nam ing services as a tool for
associating m u ltip le interfaces w ith d evices w hen N N M cannot d o so
otherw ise via SN MP. WIN S-based nam ing is su p p orted , bu t generally
is not a best p ractice, since Microsoft itself is m oving aw ay from it in
favor of DN S. If the WIN S server is not close to the N N M server, a
netw ork ou tage cou ld rend er N N M p ractically u nu sable if it can‟t
reach the WIN S server.
The sam e is tru e of DN S servers, bu t DN S is au to m atically cached
locally u nd er Wind ow s and caching DN S is often recom m end ed for
UN IX N N M servers. DN S is the p referred nam e service for N N M.
Other nam ing services have trou ble w ith the situ ation w here a rou ter
cannot be ad d ressed by SN MP and N N M w ou ld otherw ise create
sep arate nod es for each interface since it has no other w ay to know the
interfaces belong to the sam e d evice.
DN S can p rovid e the m ap p ing of m u ltip le interfaces to a single nod e
nam e. H osts files can‟t d o this, nor can WIN S.
How node objects are named
Objects are assigned IP H ostnam es based on w hether netmon can find a
nam e by d escend ing throu gh the follow ing checks:
 A non-m igratable uniquely-nam ed software loopback
interface (not 127.0.0.*). Determ ined by querying
ifType for softwareLoopback
 gethostbyaddr results on Preferred IP (see below )
 SN MP sysN am e returned on Preferred IP (see below )
24
NNM and Name Services
 Preferred IP:
 Low est num bered IP Add ress
 Low est Softw are loopback IP ad d ress (not 127.0.0.*)
 N AT ad d ress (introd uced in V6.4x)
 Low est non-m igratable IP Add ress
 Use the IPX server ad d ress (Window s only)
 Use "low est" IPX ad d ress (Window s only)
 LLa add ress ( MAC Ad d ress )
 A m igratable nam e
 The current nam e (if present in DB)
 The NN M UUID
The best p ractice is to assign softw are loop back ad d resses w herever
p ossible. The ad d ress that resp ond s to SN MP and is the p referred
ad d ress/ interface for m anagem ent, shou ld
be given the
A record in DN S, and a corresp ond ing PTR in its reverse zone. If other
conventions are u sed , there m ay be a m ism atch in a nod e‟s nam e and
the nam e u sed by N N M for trap s from that nod e. In all cases, reverse
looku p s are heavily u sed by N N M so PTR record s shou ld alw ays be in
p lace for all interfaces that N N M can‟t associate w ith a nod e throu gh
SN MP. The PTR‟s for all other interfaces bu t the one u sed for
m anagem ent, shou ld be a stand alone PTR p ointing t o the FQDN
configu red in the A-record
Assigning softw are loop backs has benefits other than for N N M alone.
Using a loop back interface in OSPF can m ake OSPF netw orks m ore
stable. Cisco in fact recom m end s u sing them . Other u ses for loop backs
inclu d e load balancing betw een BGP p eers over tw o or m ore interfaces
on Cisco rou ters. Cisco Works u ses m u ch the sam e ru les as N N M for
selecting a p referred SN MP ad d ress
SN MP sysnam e for any d evice can be w ritten from the N N M server if
an SN MP w rite com m u nity is enabled for the d evice. This can be d one
from the SN MP MIB brow ser or from the com m and line u sing:
snmpset –c <writeString> <target> sysName.0 octetstring
<nameString>
N ote: Loop back ad d resses on rou ters in the sam e su bnet can cau se
rou ting to not w ork p rop erly. This is avoid ed by u sing 32 bits m asks.
25
Fognet’s Field Guide to OpenView NNM
If the rou ters su p p ort it and they are ru nning a classless rou ting
p rotocol, a fu ll 32-bit m ask for the loop back IP can be u sed to save on
IP ad d ress sp ace. This w orks fine w ith any version of N N M after 6.2,
and Cisco rou ters w ith p ost 12.3 IOS. On p rior versions of N N M, the
follow ing errors m ay be seen: From load hosts: "WARN IN G: Invalid
broad cast IP Ad d ress 255.255.255.255, ignoring entry for <target>."
From the GUI, the error is: "The IP ad d ress is the broad cast ad d ress for
this netw ork." For m ore on this, see the section on 31 and 32-bit
netm asks on p age 47.
It is im p ortant to d raw the d istinction betw een nam es and labels in
N N M. N am es are things that m u st be u niqu e in the object d atabase
(ovwdb) and top ology (ovtop m d ) d atabase. In N N M, nod es have tw o
kind s of nam es: IP H ostnam es and Selection N am es. Labels are the
strings that show u p on the nod e in a m ap .
How preferred IP address for SN MP is chosen
The p referred SN MP ad d ress for a nod e is chosen accord ing to the
above p roced u re. This is the ad d ress u sed to access the SN MP agent on
the nod e.
The p referred IP ad d ress u sed can be m anip u lated u sing the op tions
listed on p age 28.
How node objects are labeled
N od e selection nam es are by d efau lt the sam e as the IP H ostnam e,
w hich is d eterm ined accord ing to the ru les above. Users (and third
p arty ap p lications) can change the selection nam e. If the selection
nam e for tw o objects conflict, a nu m eric ID string is ap p end ed to one
of the selection nam es in ord er to achieve u niqu eness. N od e labeling
ru les, in d escend ing ord er of p reced ence, are:
 If the nod e has an IP hostname, truncate it to the basenam e (strip off
d omain and subdomain nam es)
 If the nod e has a N etWare server nam e, use it, otherw ise, use the netw ork
num ber of the internal server add ress
 If the nod e reports an SN MP sysNam e value, use it, truncating after any
blanks in the returned value
 If the nod e supports IP, the IP address is form atted as string
 If the nod e supports IPX, the host-ad d ress portion is used
 If the nod e has a LLA/ MAC, th e physAd d r is form atted as a string
26
NNM and Name Services
In environm ents w here the u se of SN MP sysN am e is p referred over
any nam es assigned to the d evice in nam e services, u se the
ip N oLooku p .conf, d escribed on p age 28.
How interface objects are named by net mon
In N N M 6.2 and later, if the agent su p p orts the ifAlias MIB variable
and it is non-blank and u niqu e on the nod e, it shou ld be u sed ,
otherw ise a non-blank resp onse for ifN am e is u sed . If this isn‟t
available, ifDescr w ill be u sed , bu t only the first p art u p to any blanks
(u p to V7.51 Interm ed iate Patch 18, see section below ). To force the
u se of ifN am e/ ifDescr only, u se the netmon LRF setting
"-k u seIfAlias=false". ifAlias is fou nd in :
ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifAlias
If a d u p licate ifAlias is d etected for a nod e, N N M generates an alarm .
Either resolve the d u p licate nam ing issu e (p referred solu tion) or force
ifN am e globally w ith the netmon LRF sw itch m entioned above. See
p age 5 for instru ctions on u sing the LRF p roced u re.
When changing the ifAlias, or w hen changing the LRF sw itch, a
d em and p oll to the nod e m ay or m ay not p rop erly u p d ate the object‟s
label. There w as a know n p roblem w ith N N M 6.2 and ifAlias changes
that w as ad d ressed via p atches. If it d oesn‟t u p d ate p rop erly, the nod e
shou ld be d eleted from all m ap s and red iscovered .
ifD escr naming restrictions
N N M V7.51 Interm ed iate Patch 18 ad d ed su p p ort for w hite sp aces in
the ifDescr SN MP variable. Prior to this, the text after the first w hite
sp ace w as tru ncated . To allow the com p lete text of ifDescr to be u sed ,
set the follow ing environm ental variable to a valu e of “tru e:”
OV_N N M_IFLABEL_IFDESCR
How netw ork objects are named
When a new netw ork object is d iscovered by netmon, the netw orks file
or DN S is searched for m atching nam es. If non e are fou nd , the IP
netw ork ad d ress for the netw ork is u sed . The files are:
Wind ow s:
UN IX:
\ w indow s\ system32\ d rivers\ etc\ netw orks
/ etc/ netw orks
27
Fognet’s Field Guide to OpenView NNM
To change the nam e of an existing netw ork nam e in N N M after
u p d ating the netw orks file, ru n:
ovtopofix –u <previous name for network object>
N ode and interface naming restrictions
The only allow able characters in a label for a host nam e in N N M are
ASCII letters, d igits, and the d ash character. The u nd erscore character
is illegal. In essence, N N M, like m any other ap p lications, enforces p reRFC 2181 ru les w ith resp ect to allow able characters in hostnam es.
More recent versions of N N M m ay have relaxed som e of these
restrictions.
Labels m ay not be all nu m bers, bu t m ay have a lead ing d igit (e.g.
3com .com ). Labels m u st end and begin only w ith a letter or d igit, to a
m axim u m of 63 characters. Letters are case-insensitive, thou gh p reN N M 5.0 versions p reserved case. N N M u ses this convention based on
RFC1034 section 3.5.
Less stringent nam ing conventions ap p ly to interfaces, how ever.
Problem s m ay arise from the u se of certain characters in ifN am e and
ifAlias entries, and interface nam e lengths are lim ited to 63 characters
as w ell.
ipN oLookup.conf, snmpnolookupconf, excludeip.conf
ip N oLooku p .conf is a configu ration file w hich sp ecifies IP ad d resses
against w hich looku p s w ill not be p erform ed .
snmpnolookupconf is a com m and line u tility w hich m aintains the
SN MP N o Looku p cache, w hich is p art of the SN MP configu ration
d atabase (ovsnm p .conf). IP hostnam e entries in the N o Looku p cache
w ill not have nam e service qu eries issu ed to them .
If the above tools d o p rovid e the d esired resu lts, for exam p le, to
ad d ress nam ing issu es stem m ing from the low est IP ad d ress being
u sed , u se the exclu d eip .conf configu ration file to m anip u late the
selection of the p referred IP ad d ress.
General D N S considerations
netmon d iscovers IP ad d resses first. To d eterm ine hostnam es, netmon
p erform s reverse looku p s on the IP ad d resses u sing the gethostbyaddr
system call. Reverse looku p s in DN S-configu red environm ents w ill fail
u nless PTR record s are exp licitly d efined w ithin DN S. Failu re to
28
NNM and Name Services
p op u late PTR record s in DN S is one of the m ost com m on cau ses of
nam e service related issu es in N N M.
To u se DN S to force a p articu lar ad d ress to be chosen over w hat N N M
d efau lts to (often the nam e associated w ith the low est nu m bered IP
ad d ress), u se the follow ing p roced u re. Create an ord inary
A-record for the p rim ary interface w ith a corresp ond ing PTR-record in
the reverse zone. For all other IP ad d resses, one can m ake another
nam e record (A-record ) for the nod e, w ithou t a PTR p ointing back. Bu t
for all those ad d resses, m ake a PTR p ointing at the A -record for the
p rim ary nam e.
Preferrably, in the PTR record s, take each of the nod e‟s IP Ad d resses
and p oint them back to the com m on A -record . Then alias the other IP
ad d resses in the forw ard looku p s w ith CN AME record s. If the reverse
looku p d oes not m ap back to the A-record , then N N M m isses
the connection back to the nod e w hen an SN MP trap d oes not com e
from the A-record ‟s IP ad d ress.
Cache-only D N S
As m entioned above, there are several reasons w hy a cache-only DN S
server shou ld be consid ered on the N N M server. The m ain reason is to
p rovid e local control of d evice nam ing for netw ork m anagem ent
p u rp oses. A local cache-only DN S u sed to be m ore frequ ently
recom m end ed for N N M server p erform ance im p rovem ents, bu t H P
m ad e som e red u ctions to N N M‟s u se of looku p s as w ell as ad d ed
som e levels of u ser control over looku p behavior (see above). Still, a
local DN S server w ou ld increase looku p sp eed and red u ce netw ork
load , as N N M is still a heavy u ser of DN S.
Another com p elling reason for a local DN S server is to elim inate the
scenario w here the netw ork link betw een the N N M server and the
DN S server are severed . N N M fu nctions are severely ham p ered by the
loss of nam e services d u ring norm al op erations. N N M p rovid es som e
caching of looku p d ata throu gh the SN MP configu ration cachedb,
w hich is p art of the ovsnm p .conf d atabase. This d oesn‟t cache all
looku p d ata, how ever, p articu larly d iscovery -related looku p s.
N ote also that Wind ow s 2000 and above servers (and w orkstations)
p rovid e a DN S client that acts as a cache-only DN S server, bu t this
d oes not p rovid e any configu rability. It d oes bring into qu estion the
reason to install a cache-only DN S server on N N M servers solely for
29
Fognet’s Field Guide to OpenView NNM
p erform ance
how ever.
im p rovem ents
or
to
p revent
service
d isru p tions,
Split horizon D N S
Sp lit H orizon DN S configu rations m ay offer greater flexibility and ease
of control for netw ork m anagem ent p u rp oses over cache-only DN S. In
general, there are three architectu res for sp lit horizon DN S: Tw o
sep arate content DN S servers, each w ith d ifferent d atabases; a single
content DN S server serving u p m u ltip le d atabases; and a single
content DN S server serving u p a single d atabase w hose record s are
tagged for visibility to sp ecific clients.
With N N M, any of above op tions can be u sed to achieve finer control
over netw ork m anagem ent nam ing w hile not com p rom ising netw ork
ap p lications or secu rity. In fact, netw ork m anagem ent is often the
p rim ary reason for u sing Sp lit H orizon DN S.
Another com m on p ractice in this regard is to configu re a local DN S for
a su bd om ain of d evices, w here that DN S server is then au thoratative
for d evices only of interest to the netw ork m anagers and caching for all
other d evices.
BIN D -based D N S implementation troubleshooting
The first versions of BIN D 9 broke the ability to coerce sortlist ord er for
resou rce record sets that w as p ossible in BIN D 8 u sing the RRSET
op tion show n below . Drop p ing RRSET m eant that rand om -cyclic
behavior d eterm ined w hich DN S record w ou ld be retu rned first in the
list, w hich is w hat N N M u ses to d eterm ine IP H ostnam e. The op tion
w as re-instated in BIN D 9.2.3. The BIN D 8 and BIN D 9.2.3 (or later)
m ethod for fixing retu rn ord er w ithin the op tions block in nam ed .conf
is:
rrset-order {class IN type A "hostA.local" order fixed;};
The follow ing nam ed .conf logging op tions are u sefu l for tracing w hat
DN S qu eries are being generated by N N M:
logging {
channel "queryfile" {
file "/var/log/dns-query.log" versions 4 size
5m; # 20MB rolling logs
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
category "queries" {
"queryfile";
30
NNM and Name Services
};
};
BIN D -based D N S on Window s
Microsoft‟s bu ilt-in caching “DN S Client” service m ay not p rovid e
enou gh flexibility and control, as it is not at all u ser -configu rable.
Microsoft‟s DN S Server is also d ifficu lt to w ork w ith (accord ing to
som e). ISC BIN D is easy to install on the N N M server as a cache-only
DN S server that p rovid es fu ll control, and is free. Be aw are, how ever,
that BIN D, like SN MP, has been the su bject of secu rity vu lnerabilities
in the p ast. More info can be fou nd on this throu gh SAN S.
The follow ing step s ou tline configu ring ISC BIN D on Wind ow s:
1.
2.
3.
4.
5.
6.
7.
8.
9.
Dow nload BIN D binary and d ocs from w ww .isc.org
Extract and install package
Run: rndc-confgen –a
Create configuration files; see exam ples in d ocs or on w eb
Check configuration files w ith namedcheckconf
Start ISC BIN D service and monitor App Log events
Test forward and reverse lookups w ith nslookup
Disable Window s DN S Client Service
Reboot and check event log again
N ote: When m aking m ajor changes to nam e services that the N N M
server su bscribes to, alw ays exp ect som e issu es w ith N N M‟s top ology
and general stability. Often, it is ad visable to start d iscovery from
scratch is su ch cases. See p age 12 for this p roced u re.
“N ame services are performing poorly” alarms
This OpenView enterprise event is generated when NNM’s internallygenerated lookups are taking too long or constantly timing out. If nodes
are being polled via netmon, the following command may provide some
information about lookup performance:
ovstatus –v netmon
See the appropriate section below to address performance issues with
name services, and also refer to the NNM “Managing” guide in Appendix
E and other sections.
31
Fognet’s Field Guide to OpenView NNM
Tuning lookup performance (Window s)
N N M u ses gethostbyname and gethostbyaddr system calls, bu t
d efau lt N T settings for these requ ests can be very long. By d efau lt, N T
shou ld p erform a N etBIOS looku p first, then a hosts file
(\ w ind ow s\ system 32\ d rivers\ etc\ lm hosts) looku p . A N etBIOS
looku p tim eou t is 4.5 second s, based on the follow ing registry valu es:
H KLM\ System\ ControlSet001\ Services\ N etBt\ Param eters:
N am eSrvQueryTim eout=1500 m sec
N am eSrvQueryCount=3
If DN S or WIN S is configu red , typ ically these m ight be called on first,
and then N etBIOS and hosts looku p s w ou ld be p erform ed . Make su re
"Enable DN S for Wind ow s Resolu tion" is checked , and "Enable
LMH osts Looku p " is u nchecked .
The recom m end ed registry p aram eter settings are as follow s:
H KLM\ System\ ControlSet001\ Services\ N etBt\ Param eters:
Set Nam eSrvQueryCount to 0
Set Nam eSrvQueryTim eout to 100 to 500
Set Nod eType to 2 ( i.e. P-nod e )
N am eSrvQu eryTim eou t shou ld be set based on the overall latency in
the netw ork. N ote that w ith these settings, access to other N T servers
d oes not ap p ear to be affected , how ever access to LAN Manager
servers d oes.
Control the search ord er u sing the follow ing keys u nd er:
H KLM\ System\ CurrentControlSet\ Services\ TcpIP\ ServiceProvider:
DnsPriority
LocalPriority
H ostsPriority
N etbtPriority
These shou ld have valu es assigned to them the range of -32768 to
32767. The lower the nu m ber the higher the p riority. The p riority ord er
d eterm ines the ord er that they are u sed . localPriority affects the nam e
looku p u sing the lm hosts file.
There is also another key:
32
NNM and Name Services
H KLM\ System\ CurrentControlSet\ Services\ TcpIP\ Param eters\ DnsN btLookupOrd er
This affects w hether DN S has p riority over N etBIOS. A valu e of 0
ind icates DN S has p riority, a valu e of 1 ind icates N etBIOS d oes. See
MS d ocu m ent Q120642 for m ore inform ation. A best p ractice is to
sim p ly d isable N etBIOS over TCP/ IP (N etBT). If this is d one, it shou ld
only com p rom ise file sharing to/ from N T 4.0 servers.
If not d isabling the N etBT, set the N N M server as an H -N od e or PN od e for N etBT resolu tion. This w ay, if a nam e isn't resolved by one of
the other m ethod s, the server shou ld d o a qu ery d irectly to a WIN S
server, w hich is m ore likely to su cceed on a large netw ork than a
broad cast. A P-N od e d oes not d o a broad cast before giving u p , bu t is
not allow ed on som e netw orks. This is set in:
H KLM\ SYSTEM\ CurrentControlSet\ Services\ Tcpip\ ServiceProvid er
w ith the key nam e being Class. A valu e of 8 m akes the com p u ter an
H nod e. The Class valu e for B nod e is 1, for P nod e is 2, and for
M nod e is 4. Keep in m ind that m ost Wind ow s DN S servers are
configu red to d o a WIN S qu ery them selves if a na m e or ad d ress isn't in
their zones. WIN S nam e resolu tion failu res can be m ad e to fail faster
by ad ju sting:
H KLM\ SYSTEM\ CurrentControlSet\ Services\ N etBT\ Param eters:
BcastNam eQueryCount
BcastQueryTim eout
N am eSrvQueryRetries
N am eSrvQueryTim eout .
If Active Directory is involved Wind ow s u ses the DN S servers know n
to Active Directory first, even if they are d ifferent then the configu red
DN S servers. To sw itch to a locally-controlled DN S server, get the
N MS Server ou t of the Active Directory tree and p oint it at a DN S
Server that is not know n to Active Directory, p referably on the sam e
su bnet. Active Directory is not requ ired to ru n N N M..
Tuning lookup performance (UN IX)
The m ost im p ortant configu ration entry p oint for UN IX nam e services
tu ning is the / etc/ nssw itch.conf file. This file sets the ord er and
failover behavior for several services inclu d ing nam e services.
The nslookup com m and op erates d ifferently on H P-UX and Solaris.
On H P-UX, nslookup qu eries DN S, N IS, or hosts; on Solaris, nslookup
33
Fognet’s Field Guide to OpenView NNM
qu eries DN S only. On Solaris, u se getent hosts <target> qu eries to
test how looku p s are affected by nssw itch.conf.
On H P-UX and Solaris, DN S tim eou ts can be tu ned d irectly in the
/ etc/ resolv.conf file. By d efau lt, the tim eou t is calcu lated u sing the
RES_RETRAN S p aram eter w hose d efau lt is 5000 m illisecond s (H P -UX)
and 5 second s (Solaris) and RES_RETRY w hose d efau lt is 4.
The follow ing entries lim it the tim eou t 1s p er failed looku p :
domain <domain>
nameserver <primaryNameServer>
nameserver <secondaryNameServer>
retrans 1000 # value in milliseconds
retry 1
N N M doesn’t detect D N S changes (Window s)
With Wind ow s 2000, Microsoft introd u ced a local caching -only DN S
server that gets qu eried first for any DN S looku p s, regard less of
looku p ord er. This is the DN S Client service, and it is installed and
ru ns au tom atically by d efau lt.
Od d valu es cou ld still be retu rned from the local client after another
DN S server is u p d ated . Disabling the DN S Client service if ru nning a
caching-only DN S on the N N M server, is recom m end ed on the N N M
server as N N M alread y caches qu eries via the SN MP Configu ration
d atabase. That cache can be cleared u sing:
xnmsnmpconf -clearC
To d u m p and then clear the contents of th e local DN S cache , ru n:
ipconfig /displaydns
ipconfig /flushdns
To A record or not to A record
N ew er versions of N N M have m ore intelligence and flexibility for
p icking the p referred m anagem ent IP ad d ress (aka p rim ary IP
ad d ress). For p rim ary IP Ad d resses, alw ays ad d an A record and a
PTR record for the resverse zone.
Ad d ing A record s for second ary IP ad d resses is generally not
necessary for N N M‟s p u rp oses, bu t m ay be d esirable for other reasons.
When ad d ing A record s for second ary IP ad d resses, instead of ad d ing
a PTR record p ointing back to that A record ‟s nam e, u se the nam e
associated w ith the A record for the p rim ary IP ad d ress.
34
NNM and Name Services
A record compare issue on Solaris platform
Solaris im p lem ents sp ecial logic in gethostbyaddr su ch that after the OS
requ ests the PTR from the DN S and gets the A record back, it issu es a
su bsequ ent requ est for the A record and com p ares the resu lts to the IP
ad d ress originally requ ested . If this ad d itional requ est isn't retu rned ,
the w hole call retu rns N ULL. This logic w as institu ted as IP sp oofing
p rotection.
This can be p roblem atic on m u lti-hom ed hosts that d o not have
A record s for every IP ad d ress d efined for the nod e in the case w here
A record s cannot otherw ise be entered for all interfaces in DN S d u e to
issu es w ith rou nd robin, ap p lication requ irem ents, etc. Work -arou nd s
inclu d e only entering the loop back ad d ress for the d evice in DN S or
u sing a com bination of noIp Looku p .conf and snmpnolookupconf. See
p age 28 for m ore on these settings.
nscd issues on Solaris
The N am e Server Cache Daem on, nscd, is intend ed to sp eed u p nam e
service looku p s, w hether from N IS, N IS+, DN S, or local files. In som e
cases, how ever, nscd can slow DN S looku p s, and som e ad m inistrators
choose to d isable it. Moreover, nscd interferes w ith rou nd robin (nscd
caches record s in one ord er and d oesn't rotate them ). Som e im p ortant
/ etc/ ncsd .conf file tu nables are:
Toggle host lookup caching:
TTL positive results:
TTL negative results:
enable-cache hosts <yes | no>
positive-time-to-live hosts <sec>
negative-time-to-live hosts <sec>
N ame resolution support commands:
Use the follow ing com m and s to d eterm ine if N N M is p erform ing p oorly
w ith resp ect to nam e services:
$OV_SUPPORT/gethost <hostname>
$OV_SUPPORT/gethost –v <host> (fw d and reverse lookup)
$OV_SUPPORT/gethost –a <IP Address>
xnmsnmpconf –resolve <target>
xnmsnmpconf –dumpcache all
xnmsnmpconf –clearC
snmplookupconf -dumpC
snmplookupconf –t[est] <target>
$OV_SUPPORT/checkDNS.ovpl
ovstatus –v snmpCollect
ovstatus –v netmon
35
Fognet’s Field Guide to OpenView NNM
N N M D N S tracing
Set the follow ing environm ent variable to enable DN S tracing to the
sp ecified file, then restart the Op enView d aem ons:
OV_NS_LOG_TRACE=<directory>;0.0;2;
See p age 3 for inform ation on setting environm ent variables. Logfile
entries have the follow ing form :
N:2:N:...
N:2:A:...
H ostnam e to IP Ad d ress lookup
IP Ad d ress to H ostnam e lookup
N ame services-related documentation
N N M 6.2 p atches introd u ced DN S p erform ance im p rovem ents w hich
are d iscu ssed in the follow ing w hite p ap ers:
H P‟s White Paper: $OV_DOC/ WhitePapers/ DN Sand OpenView .pd f
D. Stevenson‟s DN S White Paper: w w w.fognet.com / d s-d ns.doc
N N M in demo, D HCP or mobile environments
From an SNMP management perspective, the NNM server needs
to be a fixed IP address that doesn‟t change often. Changing the
IP address of the NNM server causes SNMP devices to generate
traps to all the addresses that NNM discovers them from, because
the NNM server sets itself as a trap destination when it can.
If NNM is to be installed on a laptop that is moved around, or it
is a demo system where the IP address is expected to change, set
the USE_LOOPBACK=ON parameter in the following file to keep
NNM daemons from crashing when the network becomes
unavailable: $OV_CONF/ov.conf. Typically, in this scenario,
the NNM license is instant-on, otherwise it would report an error
if the licenced IP changed.
36
5. Discovery via netmon
N N M has tw o d iscovery engines and tw o p olling engines, each of w hich
ad d ress OSI level 2 and level 3 resp ectively, and each of w hich has
overlap onto the other layer. netmon is the level 3 d iscovery and p olling
engine and is a legacy p rod u ct elem ent. ovet_disco and ovet_poll are the
new er level 2-focu sed d iscovery and p olling d aem ons.
The ET p ollers d ep end on netmon for d iscovery, and the ovet_bridge
p rocess is resp onsible for keep ing the sep arate top ology d atabases
synchronized . Distribu ted N N M architu re is bu ild m ostly on legacy level
3 top ology and d iscovery, and ET d iscovery is not necessarily easily
com p atible w ith d istribu ted installations.
When N N M is first installed , netmon d iscovery occu rs au tom atically
u nless d isabled d u ring d ialog at installation. To get layer2 top ology
d isovered , the ovet_disco-based d iscovery m u st be enabled u sing the
follow ing com m and :
setupExtTopo.ovpl
Sim ilarly, the ovet_poll-based p oller and the ad vanced p roblem analyzer ,
if d esired , m u st be enabled u sing the follow ing com m and :
ovet_apaConfig.ovpl –enable APAPolling
Prior to ru nning ovet_disco-based d iscovery, at least som e of the
environm ent shou ld be p rop erly d iscovered by netmon first. Sim ilarly,
p rior to enabling the ovet_poll-based p oller, the ovet_disco-based
d iscovery m u st be ru n first. All netmon p olling op tions that can be set in
the Op tions-N etw ork Polling Configu ration m enu s can be controlled
from the com m and line as w ell u sing the xnm p olling com m and . For
exam p le, to toggle netmon d iscovery on or off, u se:
xnmpolling -ipDiscoveryOff or xnmpolling -ipDiscoveryOn
37
Fognet’s Field Guide to OpenView NNM
Preparing for N N M discovery
The H P d ocu m entation covers this w ell, bu t here is a review of som e
im p ortant step s to take before setting N N M loose on the netw ork:
1.
2.
3.
4.
Make sure SN MP com m unity strings are properly configured
Make sure nam e services are OK. Read the section on nam ing
Populate netmon.noDiscover file w ith und esirable d evices
Populate netmon.noDiscover file w ith H SRP d evices (skip if running
N N M 7.? or above, or use netmon.m igratable)
5.
Have any ed ge routers? Consid er –R netmon LRF sw itch
6.
Make sure firew alls are open to desired areas (see page 283)
7.
Check for cut SN MP tables on key d evices
8.
Set loopback ad d resses on routers, m ap to DN S (see page 24)
9.
Run: nmcheckconf (UNIX) to probe network configuration
10. If there are nod es w ith many interfaces that are not d esirable to
d iscover/ manage, consid er using netm on.interfaceNoDiscover
After N N M discovery
After initial au tom atic d iscovery by N N M, p erform these step s to e xp and the
d iscovered environm ent:
1.
2.
3.
4.
5.
Select d esired unmanaged n etw orks (beige) and select Map - Manage
Objects to d iscover into ad jacent netw orks
Run the follow ing and com pare to entries in oid_to_type, oid_to_sym ,
and H POid2Type (com mand is case sensitive):
ovobjprint –a ―SNMP sysObjectID‖
See the script on page 45 for a way to automate the above step
Check totals on objects d iscovered by running: ovtopodump -l
Once ET Discovery is complete, compare ovtopodump –l
output with ovet_topodump -info output
D iscovery process in a nutshell
netmon qu eries the SN MP MIB of the N N M Server to d eterm ine its IP
ad d ress, su bnet m ask, its d efau lt rou ter's IP ad d ress and its local ARP
cache. netmon read s the ARP cache of the d efau lt rou ter via SN MP,
then send s an ICMP ping to each IP ad d ress learned from the ARP
cache. See p age 49 for a list of the sp ecific MIB variables qu eried for
this inform ation.
Those nod es that resp ond to SN MP get an SN MP requ est for
sysObjectId for each d iscovered nod e, and a series of other qu eries
(listed below ) to d eterm ine other attribu tes an d cap abilities. To
38
Discovery
d eterm ine the nod e typ e, N N M m ap s the agent OID to the oid _to_typ e
or H Poid 2typ e file for vend or, SN MP agent and other top ology flags
that d eterm ine w here in the m ap the nod e object shou ld be p laced . The
oid _to_sym is then qu eried to m ap a sym bol to the nod e. Ad d itional
nam e service qu eries are m ad e and the nod e is assigned a selection
nam e accord ing to the ru les on p age 24.
D iscovery polls vs. other types of net mon polls
Discovery p olls focu s on find ing new nod es that haven‟t been fou nd
before. The d iscovery p oll looks at the rou ting table, etc., and if it find s
som ething new , netmon issu es su bsequ ent cap ability and top ology
checks, otherw ise it d oesn't.
A configu ration check or configu ration p oll com bines a cap ability p oll
and a top ology check, this com bo being the equ ivalent of issu ing the
nm d em and p oll com m and . The cap ability p oll looks at object DB
cap abilities.
The top ology p oll looks at top ology MIB tables su ch as the brid ge MIB,
etc. The top ology p oll is only issu ed to those to connector nod es, and
can be configu red to occu r m ore frequ ently in the p olling op tions.
Understanding net mon-based level 2 discovery
As m entioned above, there are som e vagaries in this legacy p oller. The
sw itches and op tions that affect netmon’s d iscovery, p olling, and
su bsequ ent top ology p lacem ent in IPMAP are nu m erou s. For the m ost
p art, it is best to leave N N M‟s L2 settings alone.
For the brave, the level2conf w hite p ap er exp lains m u ch of the
aforem entioned vagaries. Becau se it typ ically raises m o re qu estions
than it answ ers, this w hite p ap er w as rem oved from later d istribu tions
of N N M. In fact, it has been p u rged from all of H P‟s p u blic p ages, bu t
it can still be fou nd at several sites via w eb search .
Limiting discovery
$OV_CON F/ oid _to_typ e has global op tions to lim it d iscovery, for
exam p le, of all non-SN MP su p p orted d evices. All system s m atching
OID‟s p laced in that file can also be configu red to be ignored by N N M
d iscovery. Details on u sing oid _to_typ e can be fou nd on p age 7.
To d isable au tom atic d iscovery, toggle the “Discover N ew IP N od es”
rad io bu tton in the IP Discovery configu ration area of the Op tions-
39
Fognet’s Field Guide to OpenView NNM
N etw ork Polling Configu ration m enu bar item . The sam e can be
accom p lished from the com m and line w ith:
xnmpolling -ipDiscoveryOff
To lim it d iscovery by IP ad d ress range, IP ad d ress w ild card , or by
MAC ad d resses, see the m an/ ref p ages for netm on.noDiscover and
netm on.noMACd iscover. Any changes to these files requ ire that
netmon be restarted u sing ovstop and ovstart and only ap p ly to new
nod es being d iscovered .
To lim it d iscovery of interfaces based on ifTyp e, nam e, d escrip tion etc,
u se netm on.interfaceN oDiscover (available in N N M 7.5+). This w ill
p revent the interfaces from being d iscovered both by netm on and ET
d isco. See Page 53 for m ore inform ation on this.
Lim iting exp ort of nod es from netmon to ET is configu red in the
brid ge.noDiscover file in $OV_CON F. See the brid ge.noDiscover
m an/ ref p ages for m ore inform ation on this.
With the release of N N M 6.31, H P introd u ced the ability of filters to be
u sed to p revent d iscovery of w hole classes of nod es based on stand ard
N N M filters.
A d iscovery filter u ses N N M filter d efinition langu age (see p age 268)
to d iscard or inclu d e only those nod es w hich p ass the filter. For
exam p le, to d iscover only nod es w ith Cisco and Extrem e SN MP
agents, bu ild a filter like this:
DiscoInclud e “Cisco, Extrem e” {
(“SN MP sysObjectID” ~ “1.3.6.1.4.1.9.*”) | |
(“SN MP sysObjectID” ~ “1.3.6.1.4.1.1916.*”)
}
Or to exclu d e Microsoft nod es from d iscovery:
DiscoExclud e “Microsoft: {
{ isN od e && ( vendor != "Microsoft" )
}
Ru n ovfiltertest to see the resu lts of the filter if any of the d evices to
inclu d e or exclu d e have alread y been d iscovered .
To activate the d iscovery filters, u se the m enu bar p u ll d ow n:
Op tions:N etw ork Polling Configu ration:IP. Select Configu ration Area:
40
Discovery
General. Select Use Discovery Filter, then enter the filter nam e , for
exam p le, DiscoExclu d e.
The follow ing external com m and s can be u sed to d ep loy and control a
d iscovery filter nam ed <filter-name>:
xnmpolling –discFiltName <filter-name> -discFiltOn
xnmpolling –discFiltName <filter-name> -discFiltOff
Once the filter is configu red , tested , and ad d ed to the p olling
configu ration, all nod es to be exclu d ed m u st be d eleted from the
top ology and all existing m ap s m u st be op ened . Once they are d eleted ,
they w ill not be red iscovered .
Externally managing or unmanaging objects
Objects are typ ically m anaged or u nm anaged u sing m ap op erations in
the GUI, bu t here are fou r com m and s u sed to m anage or u nm anage
objects externally (in increasing ord er of com p lexity):
xnmtopoconf
ovtopofix
topology URLs
ovautoifmgr
See p age 233 for inform ation on ad d ing, d eleting, m anaging or
u nm anaging objects via top ology URLs. N ote that none of the above
tools can be u sed to control d iscovery by ET or p olling by APA. In
N N M 7.5 and above, u se ovet_toposet.ovpl to u nm anage nod es,
board s and interfaces from being statu s p olled by the APA. More
inform ation on ovet_toposet.ovpl can be fou nd on p age 75.
xnmtopoconf can be u sed to m anage or u nm anage ind ivid u al nod es
from the com m and line. xnmtopoconf is only available on N N M
ed itions su p p orting DIM (i.e. not N N M Stand ard Ed ition). Use
ovtopofix –G/g to m anage/ u nm anage objects by OOID. For
exam p le, get OOIDs for all interfaces w hose nam e is Se0/ 1.1, then p ass
to ovtopofix –G:
ovobjprint -a "SNMP ifName" "SNMP ifName"="Se0/1.1"
The follow ing tw o com m and s, resp ectively u nm anage (-g) and m anage
(-G) objects:
ovtopofix –g <OOID>
ovtopofix –G <OOID>
41
Fognet’s Field Guide to OpenView NNM
Use ovautoifmgr to au tom atically m anage or u nm anage interfaces based
on filters. For exam p le, an ad m inistrator m ay w ant to m anage only
interfaces w ith IP ad d resses assigned on sw itches. First, set u p tw o
filters for the ovau toifm gr.conf file, one that p assed the nod es w hose
interfaces are to be scanned by the second filter. In the filters file,
d efine tw o filters:
Switches "Switches Node Filter" { isBridge }
NonIpInt "Non-IP Interfaces filter" { !isIP }
Then, in the ovau toifm gr.conf file, ad d : Switches NonIpInt
Stop discovery of non-IP interfaces for a device class
Use the –N flag in the oid _to_typ e file. Changes affect only new ly d iscovered d evices, or ru n : ovtopofix -o <OID>. Details on u sing
oid _to_typ e can be fou nd on p age 7.
Stop all SN MP discovery/polling to a device class
On som e d evices, it is not d esirable to have SN MP visibility, bu t
sim p le ICMP statu s to the nod e is OK. Use the –I flag in the
oid _to_typ e file to force ICMP-only statu s. Changes affect only new lyd iscovered d evices, or ru n : ovtopofix -o <OID>. Details on u sing
oid _to_typ e can be fou nd on p age 7.
D iscovery hints
A com m and line op tion to p rovid e netmon d iscovery hints w as ad d ed
in interm ed iate p atch 18 to V7.51. This allow s hints to be entered
d irectly from the com m and line and can red u ce or illim inate the need
to u se seed ing (see below ) and is a m ore d irect m ethod that u sing the
loadhosts com m and (p age 45). To ad d a hint to netmon, issu e the
follow ing com m and :
netmon –h <ip addr>
Seeding discovery
Use a netmon seed file in environm ents w here red iscovery of the
netw ork is frequ ent and d evices cannot be d iscovered by netmon d u e
to netw ork or SN MP access. Entries in the seed file shou ld be
p op u lated into p reviou sly-u nd iscovered netw orks as m anaged
netw orks. This can save tim e w here otherw ise, new netw orks m u st be
m anu ally m anaged from the GUI.
42
Discovery
A seed file m ight also sp eed the d iscovery p rocess. A seed file has the
form of a stand ard hosts file and is configu red u sing the
–s <seedfile> LRF sw itch to netm on.lrf. See p age 5 for LRF
instru ctions. N ote w hen sp ecifying a file in the sw itch for Wind ow sbased servers, the colon m u st be escap ed , for exam p le:
-P -s "c\:\seedfile.txt" –k segRedux=true
Configuring multiple SN MP community strings
The netm on.cm str file su p p orts an ord ered list of com m u nities to u se
w hen contacting SN MP agents for read -only access. H ere is an
exam p le for a list of five com m u nities to attem p t to u se in ascend ing
ord er to contact all SN MP agents:
"cs1", "cs2", "cs3", "cs4", "cs5" : : : :
Preferred SN MP management address
If a configu ration p oll fails to reach a nod e‟s p referred SN MP ad d ress
as set in the SN MP configu ration DB, netmon issu es a sp ecial
“p ickSnm p Ad d rPoll” that attem p ts to find a valid alternative SN MP
p referred ad d ress. This can cau se issu es w ith d ata collections and
cou ld cau se the target‟s nod e nam e to change in N N M.
If loop backs are not w id ely im p lem ented , p ickSnm p Ad d rPoll is a
u sefu l tool to help m anage changing SN MP m anagem ent ad d resses.
Also, see p age 26 and su bsequ ent sections for tip s on u sing DN S to
u nd erstand the best netw ork nam ing conventions to m ost efficiently
SN MP m anage netw orks.
Defining loop back ad d resses on netw ork d evices also m ay obviate the
need to w orry abou t the p referred SN MP m anagem ent ad d ress
changing, since m ost netw ork d evices w ill re-assign the configu red
loop back to an active interface if the cu rrent interface fails. If this is the
case, follow the step s below to d isable p referred m anagem ent ad d ress
p icking w hich is enabled by d efau lt.
To d isable the p ickSnm p Ad d rPoll featu re, set the follow ing LRF (see
p age 5) sw itch in netmon:
-k pickSnmpAddrPolls=false
Another –k netm on lrf op tion w as ad d ed in N N M 7.53 that shou ld also
be d isabled to stop netm on from sw itching p referred m anagem ent
ad d resses. The d ifference betw een this new er one and
43
Fognet’s Field Guide to OpenView NNM
p ickSnm p Ad d rPolls is that the above op tion affects general SN MP
p olls w hile the new op tion affects netmon-based SN MP statu s p olls:
-k adjustNodeSnmpAddr=false
Forcing the SN MP address for a node
Use this m ethod if all other m echanism s for p icking the correct
p referred SN MP ad d ress fail. If u sing the APA, see p age 92 for how ET
and the APA can affect p reffered m anagem ent SN MP ad d ress. To
force N N M to change the SN MP Ad d ress to a d ifferent one, follow
these step s:
1. If using APA, d isable polling using:
ovet_apaConfig.ovpl –disable APAPolling
2. In the IPMAP top ology map, tem porarily unm anage all the interfaces for
the target nod e except the interface that is to be set as the SN MP Add ress
3. Demand Poll the d evice
4. Unless the netmon –k pickSnmpAddrPolls w as set to false, the
preferred SN MP ad d ress w ill now be set to the managed interface.
5. Run: ovtopodump –l |grep ―SNMP ADDRESS‖
6. Re-m anage the rest of the nod e‟s interfaces
7. Re-enable APA using:
ovet_apaConfig.ovpl –enable APAPolling
If there is the need to p erform this task for m ore than a hand fu l of
nod es, a com m and line tool called topodbpoke can be u sed to change
the p referred SN MP Ad d ress in the ovw top ology. topodbpoke w as
ship p ed w ith N N M 7.53 in the $OV_SUPPORT d irectory. Instu ctions
for u sing it can be fou nd in the follow in g d ocu m ent u nd er “Change
Incorrect IPv4 Managem ent Ad d ress” in this w hite p ap er:
$OV_DOC/ w hitepapers/ ETand APADeploym entGuid e.pd f
If ru nning V7.53, this d ocu m entation can also be fou nd in the Gu id e to
Using Extend ed Top ology u ser m anu al. Before N N M 7.53, the tool w as
available throu gh H P su p p ort. The au thor had a need for this tool in
October 2007 and op ened a case w ith H P, bu t H P su p p ort w as u nable
to locate the u tility. For this reason, and for folks ru nning earlier
versions of N N M, the au thor has p osted the topodbpoke cod e for all
p latform s excep t Linu x here:
w w w.fognet.com / topod bpoke.zip
44
Discovery
net mon’s auto-adjusting discovery polling algorithm
Discovery p olls are basically SN MP requ ests to rou ting table s to find
nod es that haven‟t been d iscovered before. All nod es start at 15 m inu te
intervals for new nod e p olling. If netmon d iscovers no new nod es
d u ring a p oll, then five m inu tes are ad d ed to the p olling interval. If
m ore than 5 nod es are d iscovered d u rin g a p oll, the p olling interval is
halved . When a new nod e is d iscovered , a configu ration p oll is issu ed .
Su bsequ ent configu ration p olls are issu ed every 24 hou rs by d efau lt.
PERL script: find OID s not listed in oid_to_type
This scrip t p rod u ces a list of SN MP OIDs that have been d iscovered by
netmon, bu t d o not have corresp ond ing entries in the oid _to_typ e or
H Poid 2typ e files. Consid er ad d ing the resu lting OIDs to the files and
setting d esired top ology flags for their treatm ent. N ote that on
Wind ow s, the resu lts p rod u ced are not m ad e u niqu e.
#!/usr/local/bin/perl
open IN1,"c:\openview\conf\oid_to_type";
open IN2,"c:\openview\conf\HPoid2type";
@oids1 = <IN1>; @oids2 = <IN2>;
close IN1; close IN2;
@oids = (@oids1,@oids2);
open CMD,qq(ovobjprint -a "SNMP sysObjectID" |);
@objdb = <CMD>;
close CMD;
foreach (@objdb) {
s/.*\"\.1.3.6.(.+)\"/1.3.6.$1/ or next;
chomp;
next if $oidhash{"$_"};
$oidhash{"$_"} = 1;
my @found;
foreach $x (@oids) {
push @found, $x if $x =~ /^$_/;
}
print "$_ not in oid_to_type\n" unless @found;
}
loadhosts
The loadhosts com m and is u sefu l for ad d ing nod es to the N N M
top ology ou tsid e of the d iscovery p rocess, bu t it can p rod u ce
u nintend ed resu lts. If ad d ing a nod e into a netw ork t hat d oes not yet
exist in the m ap , be p articu larly carefu l, since N N M m ay m ake certain
assu m p tions abou t the “classfu llness” of the netw ork.
In the p ast, loadhosts w as the p referred m ethod for forcing d iscovery
for a nod e or set of nod es that netmon cou ld n‟t otherw ise d iscover.
Since the release of Interm ed iate Patch 18 to V7.51, the best p ractice to
accom p lish this is to give netmon d iscovery hints u sing the “-h” op tion
to netmon. See p age 42 for d etails.
45
Fognet’s Field Guide to OpenView NNM
To m ake su re loadhosts honors the “classfu lness” of the netw ork of
the d evice being ad d ed . Use the “–m‖ op tion to sp ecify the netm ask
for the netw ork the d evice lies in if it is in a netw ork not alread y
p resent in N N M top ology. Still, loadhosts m ay com p lain, so it m ay be
necessary to m anu ally create a netw ork sym bol in ovw w ith the correct
m ask and then m anu ally ad d the d evice u sing m ap op erations. To d o
this, see p age 196.
If ad d ing less than 100 ad d resses, rem em ber t o u se the –n 1 sw itch.
Page 233 p rovid es inform ation on ad d ing, d eleting, m anaging or
u nm anaging objects via top ology URLs. This facility w as d esigned to
p rovid e a secu re rem ote front end for loadhosts. loadhosts is
d esigned to take inp u t from a file, bu t w hen invoked from the
com m and line, shou ld take stand ard inp u t. This m eans that after inp u t
is entered for the loadhosts com m and , the p rom p t m ight hang u ntil it
is interru p ted w ith a “Ctrl C” com m and .
loadhosts has m any com m and line op tions, bu t the best p ractice is to
load a host w ith m inim al sw itches and let netmon d iscover the rest. For
connectors, it is best to u se the ad d ress associated w ith the softw are
loop back to seed netmon, bu t in som e environm ents, it is better to u se
the ad d ress associated w ith the DN S nam e assigned to the d evice.
N am e services m ay be u sed by netmon to d eterm ine w hat interfaces are
associated w ith a p articu lar host if it can‟t otherw ise d eterm ine this
throu gh SN MP.
loadhosts byp asses netmon‟s norm al d iscovery sequ ence, so u p on
load ing, a new nod e m ay not receive a fu ll configu ration check. This
m eans that the icon m ay not be p rop erly rep resented on the m ap .
Force the configu ration p oll u sing the GUI, nmdemandpoll, or u se the
“-o” op tion to loadhosts to force an oid _to_typ e looku p if the d evice‟s
SN MP agent OID is listed there. loadhosts w ill not su cceed in
entering a p articu lar nod e into the top ology if the netw ork it is load ed
into is in the u nm an aged state.
Batch load hosts having a su bnet m ask of 255.255.255.248, w here the
file contains an / etc/ hosts-like list of IP ad d resses and hostnam es:
loadhosts –bvpm 255.255.255.248 <filename>
To ad d a single nod e via p ip ing (w orks on Wind ow s, too):
echo ―192.168.1.9 patchy‖ | loadhosts –V –v –m 255.255.255.0
46
Discovery
loadhosts op erations m ay fail for a variety of reasons. A typ ical error
m essage m ight say that the su bnet m ask is invalid . The m ost com m on
cau se is that there is an incorrectly classed netw ork d efined (see p age
203). There are other reasons loadhosts m ay fail, su ch as firew alls, etc.
In this case, ad d the objects m anu ally u sing the p roced u re on p age 196.
In general, if the object can be reached via ICMP or SN MP, it can be
load ed into the top ology. Objects that can‟t be reached by either
p rotocol m ay be ad d ed , bu t can never be au tom atically connected in
the top ology or given a statu s by netmon or the APA.
D iscovery of 31 and 32 bit subnetw orks
Both 31 and 32 bit netw orks u se the netw ork ad d ress as an IP Ad d ress.
These are IP netw orks w ith a su bnet m asks of, resp ectively,
255.255.255.254 and 255.255.255.255. The N N M top ology m anager has
trou ble w ith these, since it u ses the netw ork ad d ress to n am e netw ork
level objects. To p rop erly hand le this situ ation, the -1 and -2 sw itches
w ere ad d ed as LRF sw itches to the ovtop m d d aem on. These sw itches
get arou nd the nam ing p roblem by ap p end ing / 31 or / 32 (as
ap p rop riate) to the netw ork object nam e in the object d atabase. See the
ovtop m d m an/ ref p age for m ore d etails and see p age 5 for
inform ation on m od ifying LRF files.
D iscovery of HSRP, clusters, and multilinked routers
Legacy N N M hand les m igrating or d u p licate IP ad d resses very p oorly .
N N M requ ires forcing interfaces that share the sam e IP ad d ress to go
u nd iscovered by N N M‟s IPMAP top ology by p lacing them in the
netm on.noDiscover file.
Alternatively, the d evices that ru n H SRP can be configu red w ith “D”
flag in the oid _to_typ e file, w hich assu m es that if the floating ad d ress
is not in the ip Ad d rTable, then the ad d ress shou ld go u nd iscovered .
Unfortu nately, Cisco started rep orting floating ad d resses in the active
rou ter‟s ip Ad d rTable w ith IOS 12.1(14) and greater and 12.2(13)
and greater, so the netm on.noDiscover file is the best bet .
In N N M 7.5 the follow ing tw o LRF flags w ere ad d ed to the netmon
d aem on:
-k m igrateH srpVirtualIP=true
-k d oN otDiscoverDuplicates=true
47
Fognet’s Field Guide to OpenView NNM
The first allow s the au tom atic p op u lation of virtu al grou p IP ad d resses
into the netm on.m igratable file d u ring d iscovery. The second allow s
d u p licate IP ad d resses to be p op u lated into the netm on.noDiscover file
d u ring d iscovery. Both are set to false by d efau lt.
These settings, once ap p lied , w ill only affect new ly -d iscovered d evices.
N ote that there w as a bu g w ith m igrateH srp Virtu alIP that w as
ad d ressed w ith an early p atch to V7.5.In som e cases, the u se of the
netm on.m igratable file can be u sed for clu stered nod es. H SRP virtu al
grou p IP ad d resses can also be p laced in that file.
Cisco‟s im p lem entation of m u ltilink PPP and Ju nip er‟s Mu lti-Rou ter
Au tom atic Protection Sw itch ing featu re both create active interfaces
that share a single ad d ress. In essence, netmon cannot accom m od ate the
d u p licate interface ad d ressing betw een the rou ter p airs involved
becau se both interfaces are active at the sam e tim e. Contrast this to an
H SRP-like situ ation w here a single virtu al ad d ress is only active in one
p lace at one tim e. These are best m anaged u sing the
netm on.m igratable file.
See the netmon m an/ ref p age for m ore inform ation and see p age 5 for
how to u p d ate LRF flags. See the p roced u re on p age 7 for configu ring
oid _to_typ e. See p age 243 for d etails on how ET hand les VRRP. See
p age 245 for m ore info on how ET hand les H SRP. See p age 281 for
d etails on m anaging m igratable IP ad d resses.
Summary of net mon configuration files
The follow ing external files influ ence netmon’s behavior, bu t the m ain
settings for netmon are controlled throu gh the xnmpolling and
xnmsnmpconf interfaces available from the Op tions m enu of ovw:
$OV_CON F/ polling
$OV_LOG/ netmon.trace
$OV_CON F/ oid _to_type
$OV_CON F/ H Poid2type
$OV_CON F/ exclud eip.conf
$OV_CON F/ netm on.noDiscover
$OV_CON F/ netm on.MACnoDiscover
$OV_CON F/ netm on.interfaceNoDiscover
$OV_CON F/ netm on.m igratable
$OV_CON F/ netm on.cm str
$OV_CON F/ netm on.equivPorts
$OV_CON F/ netm on.snm pStatus
$OV_CON F/ netm on.statusIntervals
$OV_CON F/ netm on.statusMapping
$OV_CON F/ nm d emand poll.ports
48
Discovery
Undocumented net mon LRF sw itch settings
See p age 5 for d etails on the LRF p roced u re.
-A
Automatically m anage netw orks w hen created .
-H 0
Don't do http d iscovery ( UN IX only )
-H 1
Do http d iscovery ( UNIX only )
-2
Don't use SN MPv2C
-k configPollDelay <#> (see below)
Im p lem enting the –H 0 and –H 1 sw itches im p roves the p erform ance
and scalability of d iscovery p olls and of configu ratio n p olls. The
equ ivalent for Wind ow s-based N N M servers is to ad d the –nohttp
and -nohttpmg sw itches to the ovcap sd .lrf file. The H TTP d iscovery
sim p ly sets a cap ability flag if the target resp ond s to a p oll to p ort 80.
The only p u rp ose of this flag is to allow searching and rep orting based
on that cap ability. It is a good general p ractice to d isable this p olling if
there is no u se for this extra d ata in the object d atabase.
V7.51 Interm ed iate Patch 18 introd u ced the -k configPollDelay <#>
op tion to allow a d elay of <#> second s for the configu ration p oll if the
state of the SN MP ad d ress is d ow n w hen the p oll initiates. This w as
introd u ced to ad d ress the case w here a system w ith m u ltip le SN MP
su bagent took som e tim e for all su bagents to start after a reboot in
ord er to p op u late the interface tables p rop erly.
SN MP queries issued by net mon
Qu eries issu ed d u ring a netmon d iscovery/ cap ability p oll. This list m ay
vary w ith N N M version:
.1.3
.1.3.6.1.2.1.1.2.0
.1.3.6.1.2.1.1.1.0
.1.3.6.1.2.1.1.5.0
.1.3.6.1.4.1.11.2.3.1.6.1.1.2.1.2
.1.3.6.1.2.1.17.1.1.0
.1.3.6.1.2.1.17.1.2.0
.1.3.6.1.2.1.17.1.3.0
.1.3.6.1.2.1.4.20.1.1
.1.3.6.1.2.1.4.20.1.3
.1.3.6.1.2.1.4.20.1.2
.1.3.6.1.2.1.2.1
.1.3.6.1.2.1.2.2.1.1
.1.3.6.1.2.1.2.2.1.3
.1.3.6.1.2.1.2.2.1.2
.1.3.6.1.2.1.2.2.1.8
.1.3.6.1.2.1.2.2.1.7
.1.3.6.1.2.1.31.1.1.1.1
.1.3.6.1.2.1.31.1.1.1.18
Test for SN MP Support
sysObjectID
sysDescr
sysN am e
hpClusterTable
d ot1d BaseBrid geAd d ress
d ot1d BaseN um Ports
d ot1d BaseType
getnext ipAd EntAd d r
ipAd EntN etMask
ipAd EntIfInd ex
ifN um ber
ifPhysAd d ress
ifType
ifDescr
ifOperStatus
ifAd m inStatus
ifN am e
ifAlias
49
Fognet’s Field Guide to OpenView NNM
.1.3.6.1.2.1.17.1.4.1.1.0
.1.3.6.1.2.1.17.1.4.1.2.0
.1.3.6.1.2.1.4.1.0
.1.3.6.1.2.1.1.6.0
.1.3.6.1.2.1.1.4
.1.3.6.1.2.1.4.21.1.7.0.0.0.0
.1.3.6.1.4.1.11.2.17.4.1.1.1.0
.1.3.6.1.4.1.11.2.36.1.1.1.1.0
.1.3.6.1.2.1.4.21.1.7
.1.3.6.1.2.1.4.21.1.8
.1.3.6.1.2.1.4.21.1.2
.1.3.6.1.2.1.4.21.1.1
.1.3.6.1.2.1.4.21.1.11
.1.3.6.1.2.1.4.22.1.3
.1.3.6.1.2.1.4.22.1.2
.1.3.6.1.2.1.4.22.1.1
.1.3.6.1.2.1.4.22.1.4
.1.3.6.1.2.1.3.1.1.3
.1.3.6.1.2.1.3.1.1.2
.1.3.6.1.2.1.3.1.1.1
.1.3.6.1.2.1.16
.1.3.6.1.2.1.16.20
.1.3.6.1.2.1.10.18
.1.3.6.1.2.1.10.30
.1.3.6.1.2.1.10.32
.1.3.6.1.2.1.10.39
.1.3.6.1.2.1.37
.1.3.6.1.4.1.9.9.23.1.3.1
.1.3.6.1.2.1.10.166.2.1.9
.1.3.6.1.3.96
.1.3.6.1.4.1.9.9.106.1.2.1.1.11
.1.3.6.1.2.1.55
.1.3.6.1.2.1.14.1.1
.1.3.6.1.2.1.15.3.1.1
.1.3.6.1.2.1.68
.1.3.6.1.2.1.17.2.1
.1.2.840.10036
d ot1d BasePort
d ot1d BasePortIfInd ex.
ipForw ard ing
sysLocation
sysContact
ipRouteN extHop
hpOVDStnType
m anagem entURL
ipRouteN extHop
ipRouteType
ipRouteIfInd ex
ipRouteDest
ipRouteMask
ipN etToMed iaN etAdd ress
ipN etToMed iaPhysAd d ress
ipN etToMed iaIfInd ex
ipN etToMed iaType
atnetAd d ress
atPhysAdd ress
atIfInd ex
rm on m ib
rm onConform ance
DS1 m ib
DS3 m ib
fram eRelay m ib
SON ET m ib
ATM m ib
cd pGlobalRun
m pls
Cisco m pls
H SRP
IPV6
OSPF
BGP4
vrrpMIB
STP
w ireless
Qu eries issu ed d u ring a netmon top ology p oll:
ifMib (RFC 1573):
.1.3.6.1.2.1.31.1.1.1.1
ifXTableifN am e
Mau MIB (RFC 1515):
.1.3.6.1.2.1.26.1.1.1.1
.1.3.6.1.2.1.26.1.1.1.2
.1.3.6.1.2.1.26.1.1.1.4
.1.3.6.1.2.1.26.4.x
rpMauTablerpMauGroupInd ex
rpMauPortInd ex
rpMauType
d ot3MauType
Rep eater MIB (RFC 1516):
.1.3.6.1.2.1.22.3.3.1.1.1
.1.3.6.1.2.1.22.3.3.1.1.2
.1.3.6.1.2.1.22.3.3.1.1.5
.1.3.6.1.2.1.22.3.3.1.1.4
50
rptrAd d rTrackGroupInd ex
rptrAd d rTrackPortInd ex
rptrAd d rTrackN ew LastSrcAd d r
rptrAd d rTrackSourceAd d rChanges
Discovery
.1.3.6.1.2.1.22.3.3.1.1.6
rptrAd d rTrackCapacity
Rep eater MIB (RFC 2108):
.1.3.6.1.2.1.22.1.4.1.1.1
.1.3.6.1.2.1.22.1.4.1.1.2
.1.3.6.1.2.1.22.1.4.1.1.3
.1.3.6.1.2.1.22.3.1.1.1.1
.1.3.6.1.2.1.22.3.1.1.1.2
.1.3.6.1.2.1.22.3.1.1.1.3
.1.3.6.1.2.1.22.3.1.1.1.4
.1.3.6.1.2.1.22.3.1.1.1.5
.1.3.6.1.2.1.22.3.1.1.1.6
.1.3.6.1.2.1.22.3.1.1.1.7
.1.3.6.1.2.1.22.3.3.1.1.1
.1.3.6.1.2.1.22.3.3.1.1.2
.1.3.6.1.2.1.22.3.3.1.1.5
.1.3.6.1.2.1.22.3.3.1.1.4
.1.3.6.1.2.1.22.3.3.1.1.6
rptrInfoTablerptrInfoId
rptrInfoRptrType
rptrInfoOperStatus
rptrAd d rSearchTablerptrAd d rSearchLock
rptrAd d rSearchStatus
rptrAd d rSearchAdd ress
rptrAd d rSearchState
rptrAd d rSearchGroup
rtprAd d rSearchPort
rptrAd d rSearchOw ner
rptrAd d rTrackGroupInd ex
rptrAd d rTrackPortInd ex
rptrAd d rTrackN ew LastSrcAd d r
rptrAd d rTrackSourceAd d rChanges
rptrAd d rTrackCapacity
Brid ge MIB (RFC 1493):
.1.3.6.1.2.1.17.1.1
.1.3.6.1.2.1.17.1.2
.1.3.6.1.2.1.17.1.3
.1.3.6.1.2.1.17.1.4.1.1
.1.3.6.1.2.1.17.1.4.1.2
.1.3.6.1.2.1.17.2.15.1.1
.1.3.6.1.2.1.17.2.15.1.3
.1.3.6.1.2.1.17.2.15.1.8
.1.3.6.1.2.1.17.2.15.1.9
.1.3.6.1.2.1.17.4.3.1.1
.1.3.6.1.2.1.17.4.3.1.2
.1.3.6.1.2.1.17.4.3.1.3
d ot1d Basedot1d BaseBrid geAd d ress
d ot1d BaseBrid geN um Ports
d ot1d BaseBrid geType.1
d ot1d BasePort
d ot1d BasePortIfInd ex
d ot1d StpPort
d ot1d StpPortState
d ot1d StpPortDesignated Brid ge
d ot1d StpPortDesignated Port.
d ot1d TpFd bTabledot1d TpFd bAdd ress
d ot1d TpFd bPort
d ot1d TpFd bStatus
51
6. Status Polling - netmon
This section covers the fu nctions w ithin netmon that d eliver statu s.
netmon is also resp onsible for d iscvoery and configu ration p olling, and
those fu nctions are d iscu ssed in Section 5, Discovery.
ICMP and SN MP idiosyncrasies and “brow n-outs”
When a nod e is not resp ond ing to p ings or SN MP requ ests, N N M
generates events and sets top ology statu s. Bu t there are other reasons
that m ay cau se ICMP or SN MP requ ests to fail that m ight p rod u ce the
sam e events and statu s in N N M w hen in fact the nod e m ay be
p erfectly healthy. Som e N N M u sers m ight change the text of statu s
events from “nod e d ow n” to “nod e not resp ond ing to p olls” to m ore
accu rately reflect this reality. The list below p rovid es ju st som e
exam p les.
 Routers d rops packets d ue to high CPU utilization coupled w ith low priority for
ICMP and / or SN MP. Assign higher priorities to ICMP and / or SN MP on routers.
Check out Cisco‟s Priority Queueing.
 N etwork congestion over WAN links is slow ing response tim es. For exam ple, if
polls are traversing fram e relay links, then check FECN , BECN , and packets
m arked DE.
 If an N N M status poll goes through a T1 link running AMI, the packet is d ropped.
This is d ue to the fact that NN M‟s ICMP payload s are set to 64 bits of all zeros, and
the old er AMI cod ing, which is intend ed for use in voice cir cuits, is d esigned to
m ake sure there is at least one 1-bit for every 8 bits to red uce potential line
harm onics. The solution is to set up the links affected to use B8ZS instead of AMI
cod ing.
 The route that SN MP or ICMP packets take to the source and from it back to the
N N M server can be d ifferent, and at least one router m ay be configured to d rop the
protocol som ew here along the w ay. Use traceroute or tracert, Problem
Diagnosis, rping, and rnetstat to d eterm ine the source of the problem .
Adjusting ICMP and SN MP status polling intervals
Both SN MP and ICMP p olling intervals are set in the SN MP
Configu ration GUI for netmon-based p olling settings. ET-based p olling
intervals and other settings are configu red throu gh XML files, the
52
Status Polling - netmon
p rim ary of w hich is p aConfig.xm l. Like netmon, ET p olling configs are
com bined for both statu s and top ology/ d iscovery.
SN MP configu ration settings are em p loyed exactly as configu red .
ICMP configu ration settings, how ever, are rou nd ed ou t the nearest
second , so the d efau lt tim eou t of 0.8 is actu ally rou nd ed ou t to
1 second for ICMP p olls. The tim eou t interval is re-d ou bled for every
retry. A configu ration of 0.8 tim eou t and 2 retries m eans that w hen a n
ICMP p oll is issu ed , it shou ld take 7 second s to fail (first p oll tim es ou t
in 1 second , first retry tim es ou t in 2 second s, and second retry tim es
ou t in 4 second s). An SN MP p oll to the sam e nod e shou ld tim e ou t in
5.6 second s.
netmon.noD iscover and netmon.MACnoD iscover files
Use these files to enter IP ad d ress and MAC ad d resses ind ivid u ally, by
ranges, or by w ild card s to p revent su bsequ ent d iscovery after
restarting netmon. Any ad d resses that have been p reviou sly d iscovered
w ill have to be m anu ally d eleted from the top ology.
To m ake su re an ad d ress is not in the top ology, u se the Find
ap p lication from the ovw m enu bar and Find by Attribu te: IP Ad d ress.
Entries fou nd can be highlighted from the Find ap p lication and then
d eleted from the Ed it m enu bar after choosing View : Select
H ighlighted .
If p op u lating the netm on.MACnoDiscover file, it m ay be of interest to
also set the –k FilterLLAOnlyN od es=tru e LRF flag to the netmon
d aem on. Setting this flag w ill force netmon to stop u sing MAC
ad d resses in the netm on.MACnoDiscover file as hints for d iscovery.
See p age 5 for how to set LRF flags.
netmon.interfaceN oD iscover file
The file $OV_CON F/ netm on.interfaceN oDiscover is u sed to d efine
how to restrict the set of interfaces to be d iscovered for a nod e. This
configu ration file w as introd u ced in interm ed iate p atch 12 to V7.5. In
V7.51, the InterfaceFiltering.p d f w hite p ap er w a s ad d ed in
$OV_DOC.WhitePap ers. In V7.53, this inform ation w as rolled into the
Using_Extend ed _Top ology.p d f u ser m anu al.
Unlike netm on.noDiscover, netm on.interfaceN od iscover w ill tru ncate
the SN MP tables u sed to p op u late interface d ata on connector nod es,
greatly im p roving d iscovery and p ollin g p erform ance. N ot only can
sp ecific interfaces be exp licitly exclu d ed (exam p le #2 below ), bu t in
53
Fognet’s Field Guide to OpenView NNM
tru ncate m od e (exam p le #1 below ), no ad d itional interfaces w ill be
su bsequ ently d iscovered once a sp ecified interface is d iscovered . Be
carefu l w ith exam p le #1 below (tru ncate) if you are not qu ite certain
that all u ninteresting interfaces w ill have a high lighted ifInd ex than the
first one m atching the exam p le. Filters can be d efined that are
exclu sive or inclu sive.
Im p ortant: netm on.interfaceN oDiscover is u sed both by netmon and
ET/ APA, so interfaces d efined in netm on.interfaceN oDiscover w ill be
d eleted both from the the nod e object in N N M and in ET. If rem oving
a large nu m ber of interfaces (thou sand s), consid er not tak ing too m any
at the sam e tim e becau se the p rocess of d eleting interfaces is very
CPU-intensive. Consid er stop p ing netm on tem p orarily, let ET finish
d eleting the interfaces and then start u p netm on again.
In the file, an asterick rep resents any nu m ber of characters u p to the
next p eriod . For exam p le, *.corp .com m atches p c.corp .com . A qu estion
m ark m atches a single character. Brackets m atch a single character,
characters in a range, or characters not w ithin a range if '!' is the first
character w ithin the brackets. For exam p le: [bf]an.fognet.com m atches
ban.fognet.com and fan.fognet.com ; [b-d ]an.fognet.com m atches
ban.fognet.com , can.fognet.com and d an.fognet.com ; and [!cz]an.fognet.com m atches only aan.fognet.com and ban.fognet.com . Use
the netmon –a 117 com m and to trace netmon’s p arsing of the
netm on.nod iscover and netm on.noInterfaceDiscover files. See p age 67
for d etails on netmon tracing. Below are som e exam p les. There are
m ore exam p les in the w hite p ap er m entioned above:
# Truncate the interface table when an interface with
# ifType of 135 is found on node 10.162.191.146
10.162.191.146 1 ifType=135
# Ignore the interfaces with ifType of 135 or 53 for
# nodes within the address range:
10.162.191.* 2 ifType=135,53
# For nodes with a name ending in "core",
# Ignoreinterfaces with ifDescr containing the string
# "VLAN" or "virtual" for node names matching ―core:
*core 2 ifDescr="*VLAN*","*virtual*"
54
Status Polling - netmon
Commonly used net mon LRF sw itch settings
There are d ozen s of LRF sw itches for netmon, each a layered bit of
ad d ed fu nctionality that has been ad d ed w ith each su bsequ ent version
of N N M. Som e sw itches are p reserved for backw ard com p atibility and
are obsolete. Som e of those p ertain to Layer 2 top ology rep resentation.
See p age 5 for d etails on u sing the LRF u p d ate p roced u re.
-b <num>
-q <num>
-Q <num>
-H
-k
-k
-k
Send burst of <num> pings, 1 per sec after retries
Increase ICMP receive queue length, see man/ ref
Increase SN MP receive queue length, see man/ ref
N ote: –q or –Q set too high m ay overload buffers
0
Disable H TTP polls to speed configuration checks
bridgeMIB=false
Red uce segm ents created from RFC 1493
snmpTimeoutImplies=unknown
Default is critical
nonIPStatusPolls=false
Red uces SN MP sw itch queries
Be su re to carefu lly read the m an or reference p age entries for these
and other netmon LRF sw itches since changing them can p rofou nd ly
change the IPMAP top ology and the granu larity of st atu s p olls.
net mon global status polling default
The global d efau lt for d iscovery p olling d efined in SN MP
configu ration is 15 m inu tes, bu t the object-based p olling introd u ced in
N N M version 6.2 overrid es this for m ost d evices of interest (rou ters,
sw itches, etc).
net mon object-based polling (V6.2+)
Object-based p olling allow s d ifferent intervals for d ifferent d evices
classes. It also allow s d ifferent intervals for p rim ary vs. second ary
interfaces as d eterm ined by netmon‟s critical p ath analysis (m ore on
this below ). Objects are d efined via N N M‟s stand ard filter d efinition
langu age, and accessed via the Poll Objects configu ration GUI,
available as a bu tton in the p olling configu ration. This GUI is a frontend to the netm on.statu sIntervals file.
In general, object-based p olling tightens the d efau lt p olling intervals
for Rou ters, Brid ges, H u bs, and loosens the d efau lt p olling intervals
for N od es to 1 hou r (V6.4+).
D ynamically-adjusting status polling by net mon
Prior to N N M V6.0, netmon statu s p olling w as static. N N M V6.0
introd u ced the first d ynam ic p olling enhancem ent to su p p ort
55
Fognet’s Field Guide to OpenView NNM
ConnectorDow n ECS and the critical p ath analysis, w here p olls to
nod es flagged as second ary failu res w ou ld be red u ced by the statu s
red u ction m u ltip lier, w hich equ als 2 by d efau lt. This m eans the p olling
interval to these nod es w ill be m u ltip lied by tw o as long as the
p rim ary nod e rem ains d ow n.
V6.31 introd u ced tw o im p ortant netmon enhancem ents that
d ynam ically ad ju st p olling intervals to su p p ort new event correlations.
These are bu ilt into netmon, so they shou ld continu e to take affect even
if those new correlations are d isabled .
This first new d ynam ic p olling change in V6.31 is if one int erface
changes on a connector, then all of that connector‟s interfaces are
im m ed iately p olled . This is non-configu rable and affects any nod e in
the object d atabase w ith the cap ability flag isConnector set. This
second new d ynam ic p olling change in V6.31 is that any interface on a
connector that has changed statu s is re-p olled accord ing to the tw o
new netmon –k sw itches below (configu red via LRF, see p age 5):
-k shortPollTime=120
-k shortPollDownCount=2
The shortPollTim e is the interval for the re-p oll, and the
shortPollDow nCou nt is the nu m ber of re-p olls to issu e. Per the
d efau lts show n above, all connector interfaces changing statu s w ill be
re-p olled at 2 m inu tes and again at 4 m inu tes after the statu s change.
net mon layer 2 polling algorithms
Statu s based on layer 2 is very lim ited in netmon w hen com p ared to
Extend ed Top ology. netmon u tilizes SN MP qu eries to Brid ge, MAU,
Rep eater and som e VLAN MIBs to d eterm ine layer 2 statu s.
Un-nu m bered interfaces are inferred from the p ort table, then p olled
via ARP. Contrast this w ith ET w hich has p rop rietary MIB extension
d ata for hu nd red s of d evices.
For netmon, in V5-V6.1, d efau lt statu s for “d ow n” p orts is Critical. In
N N M V6.2 throu gh V6.41, the d efau lt statu s for “d ow n” p orts is
Unknow n. In N N M V7.0 and above, layer 2 p olling is d isabled in
netmon and layer 2 top ology is conveyed via ET.
Becau se ET is not enabled by d efau lt in N N M 7.0 and above, there is
no layer 2 p olling of any sort tu rned on by d efau lt in these later
56
Status Polling - netmon
versions. Previou s versions, how ever, have layer 2 p olling tu rned on
by d efau lt u nd er netmon.
netmon layer 2 statu s is d eterm ined by p olling stand ard MIB2 SN MP
tables for ifAd m inStatu s and ifOp erStatu s. Defau lt SN MP statu s
m ap p ing is based on the table below and is cu stom izable via the
netm on.statu sMap p ing in N N M 6.2+. In versions p rior to V6.2, statu s
m ap p ing is fixed p er the table:
ifAd m inStatus
ifOperStatus
OV Status
---------------------------------------------------------------------dow n
any
DISABLED
testing
any
TESTIN G
up
up
NORMAL
up
dow n
CRITICAL
up
testing
TESTIN G
APA-based layer tw o statu s is entirely d ifferent. All alarm s are log only by d efau lt and top ology statu s is only reflected u nd er Dynam ic
View s m ap s. The only statu ses reflected are: Up (green), Contents
Dow n (yellow ), Unknow n (second ary failu re), and Disabled
(ad m inStatu s is d ow n) (brow n).
N on-IP unconnected port status (net mon)
The d efau lt statu s for non -IP (layer2) u nconnected interfaces
(u nconnected sw itch p orts for exam p le) is Unknow n instead of Critical
since N N M 6.2, regard less of ifOp erStatu s and ifAd m inStatu s and
netm on.statu sMap p ing file settings. To restore the p re-V6.2 behavior
set either or both of the follow ing LRF sw itches in netmon (p age 5):
-k ConnectorL2Ports=legacy
-k nonConnectorL2Ports=legacy
In N N M 7.53 Interm ed iate p atch 19, a new op tion for netm on's -k
connectorL2Ports p aram eter of "connected " w as created w hich cau ses
netmon to check for connectivity w ithin the top ology d atabase. By
d efau lt, netm on u ses brid ge forw ard ing table to d eterm ine
connectivity.
net mon SN MP-based status for nodes
The ability to p oll for p rim ary statu s via SN MP vs. ICMP w as
introd u ced to allow p olling across firew alls w here ICMP is not
allow ed bu t SN MP m ay be. It is very lim ited in its abilities and is
exclu sive – that is if a d evice is p olled for p rim ary statu s via SN MP, it
shou ld no longer be p olled by ICMP.
57
Fognet’s Field Guide to OpenView NNM
netm on.snm p Statu s is the configu ration file for d efining Level 3 IP
ad d ress ranges to p oll via SN MP.
Use the follow ing netmon –k LRF sw itch (see p age 5) to control the
behavior of SN MP p olling tim eou ts:
–k snmpTimeoutImplies=[status]
w here statu s is one of: u nknow n, u nchanged , critical (d efau lt).
D etermine net mon-based polling intervals for a device
The follow ing com m and show s the base p olling settings for a given
nod e. It d oes not rep ort on any d ynam ic p olling changes that m ight be
in effect:
xnmsnmpconf –resolve <node>
net mon critical path analysis
netmon m aintains an in-m em ory rou te to every interface. This list is
only u p d ated w hen the netmon d aem on is started , bu t a d em and p oll
(nmdemandpoll) forces a critical p ath recalcu lation for a given p ath if
issu ed to any nod e on that p ath. When an interface d oesn't resp ond ,
netmon looks at all interfaces w ithin th e critical rou te and m arks the
closest as p rim ary and all the d istance ones as second ary failu res.
ECS correlates su bsequ ent alarm s to the p rim ary failu re u sing the
ConnectorDow n correlation and netmon increases the statu s p olling
interval for second ary failu res. Failu re statu s for second ary nod es is
configu rable throu gh the p olling configu ration. The Im p ortant N od e
filter d efines nod es that are alw ays to be consid ered p rim ary by
netmon. The p ath chosen by netmon to a p articu lar d evice m ay not be
the d esirable p ath w hen there are m u ltip le p aths available. To hard cod e a p ath, u se the follow ing netmon LRF sw itch (see p age 5):
-c <critical-route-seedfile>
To d u m p the com p lete list of all critical rou te p aths m aintained by
netmon:
snmpwalk <server> .1.3.6.1.4.1.11.2.17.4.4.2.1.1.5
To d u m p verbose inform ation abou t the target
$OV_LOG/ netm on.trace, inclu d ing the critical p ath:
netmon –n <target>
58
nod e
to
Status Polling - netmon
To d u m p verbose inform ation abou t the target
$OV_LOG/ netm on.trace, inclu d ing the critical p ath:
nod e
to
netmon –i <IP>
N ode status events – interpretation
IPMAP top ology, w hich is reflected in the ovw top ology m ap s, is
hierarchical in natu re. All statu s for IPMAP top ology originates at the
interface level, that is, interface-level events are at the root of all other
statu s events reflected in the top ology. A “nod e d ow n” event really m eans
that all the interfaces w ithin that nod e are either u nreachable or u nkn ow n,
etc. The statu s colors reflected on container objects in the top ology is tied
to the below list of statu s events u sing the d efau lt statu s p rop agation ru les
as set in the m ap p rop erties. Som e of these are logged by d efau lt, m ost of
them are not, bu t th e statu s is reflected in the top ology regard less of the
logging behavior of the events.
OV_N od e_Up:
All Ifs are up or unknow n
OV_N od e_Warning:
One If is d ow n
OV_N od e_Marginal:
One If is d ow n and >1 If is up
OV_N od e_Major:
One If is up d ow n
OV_N od e_Dow n:
All Ifs are down or unknow n
OV_N od e_Unknow n:
All Ifs are unknow n
Handling multiple OV_N ode_Up events
The OV_N od e_ Up event is generated by netmon w hen it d etects that
all the nod e interfaces are u p . As a sid e-effect, w hen one of the nod e's
interfaces goes critical and after a w hile retu rns to norm al, the
OV_N od e_Up event is generated w ithou t an int erm ed iary
OV_N od e_Dow n.
This m ay not be the d esired behavior; an
OV_N od e_u p is som etim es only w anted after an OV_N od e_d ow n.
(All interfaces d ow n).
To change this behavior to there is a one-to-one correspondence
between the OV_Node_Up and OV_Node_Down events, ow these
steps:
Modify the OV_Node_Down event: (Specific 58916865) to add an action
callback to the Create_NodeDownFile.ovpl script (listed below). Send
varbind $3 and $2 to the script as respective arguments, for example:
59
Fognet’s Field Guide to OpenView NNM
OVHIDESHELL Create_NodeDownFile.ovpl $3 $2
Modify the OV_Node_UP event: (Specific 58916864) to add an action
callback to the Check_NodeDownFile.ovpl script (listed below). Send
varbind $3 and $2 to the script as respective arguments, for example:
OVHIDESHELL Check_NodeDownFile.ovpl $3 $2
Create the scripts listed below and assure they are configured as Trusted
Commands (See page 157 for details on Trusted Commands).
#!/opt/OV/bin/Perl/bin/perl
# Create_NodeDownFile.ovpl objectID [information]
# Create a file: $OV_TMP/NodeDown_objectID.txt with optional
additional information
# as contents and also appends info to the $OV_TMP/NodeDown.log
file.
# (This information can be used to store the IP-address,
downtime etc...)
use OVvars;
sub log {
my ($logstring)= @_;
open (LOGFILE, ">>$OV_TMP/NodeDown.log") ||
die ("Cannot create/open $OV_TMP/NodeDown.log\n");
print LOGFILE "$logstring\n";
close LOGFILE;
}
if
($#ARGV >= 0) {
$objectID = $ARGV[0];
$downfile = "$OV_TMP/NodeDown_$objectID.txt";
}
else {
&log ("Create_NodeDownFile.ovpl called without parameters");
exit 1;
}
open (DOWNFILE, ">$downfile") || die ("Cannot create/open
$downfile\n");
if ($#ARGV >= 1) {
$information = $ARGV[1];
print DOWNFILE ($information);
60
Status Polling - netmon
}
&log ("Created $downfile $information");
close DOWNFILE;
exit 0;
#!/opt/OV/bin/Perl/bin/perl
# Check_NodeDownFile.ovpl object_ID [information]
# Check if a file: $OV_TMP/NodeDown_objectID.txt has been
created previously.
# If present, a real OV_Node_Down has occured previously and
now a real OV_Node_Up
# message should be displayed in the Alarm browser and the file
has to be deleted to rearm
# the mechanism. The file contents and given information can be
used as desired.
# (e.g. display nodename, calculate the downtime etc.)
# Also append info to the $OV_TMP/NodeDown.log file.
use OVvars;
sub log {
my ($logstring)= @_;
open (LOGFILE, ">>$OV_TMP/NodeDown.log") ||
die ("Cannot create/open $OV_TMP/NodeDown.log");
print LOGFILE "$logstring\n";
close LOGFILE;
}
if
($#ARGV >= 0) {
$objectID = $ARGV[0];
$downfile = "$OV_TMP/NodeDown_$objectID.txt";
}
else {
&log ("Check_NodeDownFile.ovpl called without parameters,
can't check filename");
exit 1;
}
open (DOWNFILE, "$downfile") || die ("Cannot open and check
$downfile\n");
$contents = <DOWNFILE>;
if ($#ARGV >= 1) {
$information = $ARGV[1];
$message = $information;
61
Fognet’s Field Guide to OpenView NNM
}
else {
$message = $contents;
}
$category = "Status Alarms";
$localhost = "Localhost";
$message = "Node UP event for Object ID $objectID $message";
&log ("$message for category: $category ");
$cmd = "ovevent -c \"$category\" \"\"";
$cmd .= " .1.3.6.1.4.1.11.2.17.1.0.58916872";
$cmd .= " .1.3.6.1.4.1.11.2.17.2.1.0 Integer 14";
$cmd .= " .1.3.6.1.4.1.11.2.17.2.2.0 OctetString
\"$localhost\"";
$cmd .= " .1.3.6.1.4.1.11.2.17.2.4.0 OctetString \"$message\"";
`$cmd`;
close DOWNFILE;
unlink "$downfile";
exit 0;
Status event variable bindings of interest
The table below lists som e netmon interface and nod e event varbind s:
IF Status
N od e Status
Varbind #
Varbind #
Description
-----------------------------------------------------------------------------------------$2
$2
H ostnam e of nod e that caused event
$5
$5
Tim estam p event occured
$7
Interface N am e or Label
$8
IP Ad d ress of Interface or “0”
$11
Num ber of bits in the subnet m ask
* $12
Interface ifAlias
* $13
* $8
Local list of capabilities
* $14
* $9
Nam e of prim ary failure host
* $15
* $10
Nam e of prim ary failure entity
* $16
* $11
OV OID of prim ary failure entity
* $17
* $12
Description of prim ary failure entity
* $18
* $13
Prim ary failure list of capabilities
* = N ew as of N N M V6.31
62
Status Polling - netmon
For exam p le, here are tw o statu s event texts from N N M V6.31+:
IF $7 Down $12, Capabilities: $13 Root Cause $14 $15
Node Down Capabilities: $8 Root Cause: $9 $10
Contrast these w ith the sam e statu s event texts in version s p rior to
V6.31:
IF $7 Down
Node Down
ICMP burst polls
The –b <seconds> LRF op tion to netmon (see p age 5 for LRF
instru ctions) send s a bu rst of p ings, one p er second to any object after
the regu larly-issu ed set of tim eou ts and retries have been exhau sted .
ICMP redirects
If netmon –a 3 show s m essages sim ilar to this:
unexpected ICMP message 5 code 1 from 192.168.1.1
…w here the IP is the N N M Server‟s d efau lt gatew ay, the n the d efau lt
gatew ay is p robably im p rop erly configu red . This sort of error cau ses
N N M‟s p olling w aitlist to get severely backed u p . More inform ation
on ICMP red irects can be fou nd in RFC 792.
ICMP typ e 5 m essage cod es are:
0
1
2
3
Redirect
Redirect
Redirect
Redirect
datagrams
datagrams
datagrams
datagrams
for
for
for
for
the Network
the Host
Type of Service and Network
Type of Service and Host
ICMP re-d irects m ay also hap p en w hen nod es that are red irected are
on the other sid e of an internal rou ter have a rou te to a su bnet that the
hosts and are on via a non-p ersistent WAN link. When this WAN link
goes d ow n, that rou te d isap p ears from the rou ter's rou te table. So,
w hen netmon p olls the nod e, the rou ter send s an ICMP red irect to
N N M p ointing it tow ard s another server becau se the rou ter d oesn't
have the rou te to th at nod e anym ore. The other server to w hich is
red irecting is likely to be the red irecting rou ter‟s ow n d efau lt rou te,
w hich is d ifferent than the N N M server‟s d efau lt rou te.
Within N N M, the nod e m ay be rep orted d ow n becau se netmon is
trying to reach it throu gh the red irecting gatew ay‟s d efau lt rou te,
63
Fognet’s Field Guide to OpenView NNM
w hich m ay be a firew all. If this is the case, one solu tion is to have a
sep arate segm ent for the firew all from the rou ter w ith no other nod es
on it. That w ay, traffic shou ld be forced to go throu gh the internal
rou ter regard less of w hether the WAN link to the nod es is d ow n or
not.
net mon vs. APA poller
netmon p rovid es d iscovery, configu ration, and top ology as w ell as
statu s p olling. In V7.0 and later, a new APA p oller w as m ad e available
to take over netmon’s statu s p olling fu nctions, to take ad vantage of ET‟s
m ore com p rehensive p ictu re of statu s via SN MP and to get arou nd
netmon‟s lim itations d u e to legacy architectu re.
In general, netmon-based p olling is ICMP-based excep t for non -IP and
d evices sp ecifically set to be p olled via SN MP in the
netm on.snm p Statu s file. APA p olling generally com bines ICMP w ith
SN MP-based statu s p olls, is m assively m u lti-thread ed , and the d evice
typ e d ictates how the d evice shou ld be p olled . Also, the APA is
“neighbor aw are” in how it hand les failu res. The APA how ever,
requ ires the extend ed top ology is enabled and is d ep end ent on ET
d iscovery to w ork p rop erly. In som e environm ents, ET rem ains
p roblem atic to d ep loy and m aintain.
netmon Cons










Single-thread ed , single protocol, legacy issues (IP, DN S)
Polls via ICMP or SN MP, never both, SN MP polls lim ited
Second ary status d eterm ination mechanism s rud im entary
Rud im entarily d ynam ic in adjusting to neigh bor status
Cannot poll into OAD‟s; not good in hand ling H SRP, N AT, etc.
Cannot separate IP ad d ress from a physical interface
Is not “connection aw are,” rud im entary path analysis
Unique path to each interface to determ ine prim ary
Disparate configuration sw itches, files, and d epend encies
Provid es status at only tw o entity levels: interface, nod e
netmon Pros





Behavior is 100% characterizable and configurable, less FUD
Still scales w ell w hen properly tuned / controlled
GUIs available for polling custom ization / configuration
Preservation of investm ent in netmon-based status
Doesn‟t d epend on ET d iscovery
APA Cons



64
Analysis engine com plex, d ifficult to characterize/ control
APA requires ET enabing and timely ET d iscovery
Custom ization via XML files; no configuration GUI includ ed
Status Polling - netmon









netmon-based custom izations not inherited by APA, e.g.:
- Poller settings in SN MP Configuration GUI ignored
- Interfaces unmanaged in N N M topology or by ovautoifm gr still m ay be polled
- Object based polling settings based on NN M filters (APA uses ET filters)
Separate d efinition files/ filters for Im portant N od es
nm d em and poll, ovet_toposet, checkPollCfg N/ A in 7.01
Cannot be used in DIM environment MS (CS only)
Incom patible w ith LAN/ WAN Edge & MPLS SPI in 7.01 (ok in 7.5+)
IPX polling not available after sw itching to APA
Initial setup can cause flood s of events (m ostly log-only)
IPMAP topology status from APA can be problem atic
Interface count increases if form erly not m anaging Level 2
APA Pros













Multi-thread ed , m ulti-protocol (ICMP & SN MP, others)
Sw itched -topology-aware, d up IP-aware, neighbor-aware
Provid es status at m ultiple entity levels:
- Ad d ress, Interface, N od e, Connection , Board, AgPort
Provid es m ore d ynam ic polling based on queued status:
Provid es connection -oriented and d evice-oriented status
Provid es m ore accurate & tim ely status than netmon
Less reliant on ECS, more correlation at the source
Generates fewer log-only and em bed d ed status events
Polls OAD, H SRP, and other IP add ress sharing interfaces
Few er, more intelligent and m ore tim ely status events
More granularity in entity-based polling via XML files
Red uces need to use d istributed NN M
Vend or-spesific agents gives better inform ation for various types of equipm ent
net mon polling statistics
netmon p olling statistics can be view ed from the com m and line by
ru nning:
ovstatus –v netmon
Statistics can also be grap hed from ovw by selecting the N N M server
icon in the m ap and ru nning Perform ance > N etw ork Polling Statistics.
Trend s of negative nu m bers ind icate that netmon isn‟t keep ing u p w ith
its list of objects to be p olled . Very frequ ently, this cond ition on new
installations is ind icative of issu es w ith DN S looku p p erform ance.
65
Fognet’s Field Guide to OpenView NNM
Figure 6-1
The grap h in figu re 6.1 show s an exam p le trend for a from -scratch
d iscovery of a netw ork consisting of 650 interfaces. The top line
rep resents statu s p oll list length and u nd er norm al circu m stances this
line shou ld rem ain flat. This grap h w as started shortly after a fresh
installation and netmon had alread y d iscovered abou t 300 interfaces.
The second line from the top , statu s p olls in the next m inu te, abru p tly
rises then trend s back to zero.
SN MP w as m isconfigu red and the low est line, second s u ntil next
SN MP p oll w as w oefu lly behind . After that, the SN MP issu e w as
ad d ressed , and netmon w as able to d iscover another 350 interfaces. The
second s u ntil next SN MP p oll recovered nicely bu t then the second s
u ntil next statu s p oll started to trend negative, a norm al reaction to a
bu rst of new -nod e d iscovery.
The thick line that hovers below zero is second s u ntil next statu s p oll,.
Anytim e that the valu es for this m etric are below zero, it ind icates that
som e nod es are not getting p olled w ithin their d efined statu s p olling
intervals – netmon is falling behind . Und er norm al op erating
cond itions, this line stays near zero.
66
Status Polling - netmon
net mon troubleshooting, tracing and dumping
Ru n: netmon -M <mask> to send an event to the ru nning netmon to
tu rn on tracing. Once tracing is enabled , ru n: nmdemandpoll <node
name> or p erform som e other action to get netmon to p erform the task
to trace. Trace ou tp u t is w ritten $OV_LOG/ netm on.trace. Ru n:
netmon -M 0 to tu rn off tracing as netmon.trace grow s w ithou t bou nd s
and can easily fill u p a d isk w ith m ore ver bose tracem asks. Tracem asks
are ad d itive, so to trace both SN MP requ ests and SN MP rep lies and
tim eou ts, u se a tracem ask of 12 (4+8). Several tracem asks are listed in
the netmon m an/ ref p age; the follow ing are som e ad d itional
tracem asks:
0x00000100
0xffffffff
Trace effects of netmon.cm str file
Turn on all m asks – uses lots of d isk/ CPU
netmon –a <action-number> d u m p s netmon internal d ata stru ctu res
su ch as p ing lists and SN MP lists to $OV_LOG/ netm on.trace. Use the
follow ing com m and to d u m p a list of all action-nu m bers and their
m eanings: netmon –a ?
netmon –n <target> d u m p s verbose inform ation abou t the target
nod e to $OV_LOG/ netm on.trace, inclu d ing the critical p ath.
netmon
–i <IP> d u m p s verbose inform ation abou t the target
Interface to $OV_LOG/ netm on.trace inclu d ing the critical p ath.
67
7. Status Polling - APA
Introd u ced in N N M 7.01, the APA p olls H SRP and OAD by d efau lt, and
can be configu red to take over for netmon statu s p olling. Before enabling
APA p olling, read the p ros and cons of netmon vs. APA p olling on p age
64.
There w ere several changes to the APA betw een V7.01 and V7.5 and in
su bsequ ent p atches to V7.5 and V7.51. These changes concern
p erform ance, scalability, granu larity, and the ability to characterize
p olling behaviors. If u sing the APA, alw ays m ake su re the latest u p d ates
are ap p lied .
APA Architecture
Polling p olicies are labeled as Class Sp ecifications in the p aConfig.xm l
file. The three p rim ary “p aram eters” that are u ser configu rable are
snm p Enable, p ingEnable, and interval. For each object (like an ad d ress,
interface or nod e), APA begins p arsing the p aConfig.xm l from the top
d ow n looking for a p olling p olicy m atch for the object and the
p aram eter setting. It also looks for m atch for its p arent objects and
ap p lies logic to m ake a d ecision abou t how to p oll the object.
For snm p Enable and p ingEnable p aram eters, this is a logical AN D. For
interval, its m ore of an if/ else statem ent as d etail below . If it can‟t find
an exp licit p olicy m atch for the object, it reverts to the d efau lt setting.
N ote that d efau lt settings are “typ eless”, m eaning there isn‟t a sep arate
d efau lt for nod es, interfaces and ad d resses. They all share the sam e
d efau lt settings.
The APA then d oes the sam e algorithm for the p arent object and
p erform s a logical AN D of all the resu lts. In APA, N od e is the p arent
of Interface w hich is the p arent of Ad d ress.
N od e <- Interface <- Ad d ress
To find a p olling p olicy for an ad d ress in p aConfig.xm l, it w ill look
68
Status Polling - APA
for exp licit p olling p olicies for the ad d ress, the p arent interface and the
p arent nod e. It w ou ld look som ething like this:
(Ad d ress Setting) AN D (Interface Setting) AN D (N od e Setting)
So, for a given ad d ress, APA begins looking throu gh the p aConfig.xm l
file for a p olicy m atch for the ad d ress. If it d oesn‟t find one, it reverts to
the d efau lt setting w hich is PingEnable=tru e. N ow it goes to the p arent
object w hich is an interface. Su p p ose it m atches on “u nconnected
interfaces on rou ters” and that configu ration is set to PingEnable=false.
N ow it goes to the p arent object w hich is a nod e and it find s one for
rou ters and the configu ration is set to PingEnable=tru e. AN D all these
valu es together and you ‟ll get:
Tru e AN D False AN D Tru e = False
Therefore the ad d ress in not p inged . N ow , for the interval p aram eter, the
logic is sim ilar bu t rather than AN Ding the valu es together, it ju st takes
the first valu e it m atches and d oesn‟t try the d efau lt u ntil all p arents are
exhau sted . So the logic is:
(Ad d r Interval) else (Intf Interval) else (N od e Interval) else (d efault)
For exam p le, a set of rou ters has a p olling interval of 120 second s and
there are no p olling intervals set for any Interfaces and no p olling
intervals set for ad d resses. Su p p ose the d efau lt p olling cycle is set to 300
second s:
(u nd efined ) else (u nd efined ) else (120s) else (500s) = 120s
APA vs net mon status architecture
The follow ing d iagram show s the m ajor su bsystem s of both legacy
N N M and ET and their relationship s to statu s. This relates only to
N N M 7.x versions:
69
Fognet’s Field Guide to OpenView NNM
70
Status Polling - APA
APA and Smart Plug-Ins
Both the Sm art Plu g-in for Fram e Relay (version 2.0) and the Sm art
Plu g-in for MPLS IP VPN (version 1.0) are incom p atible w ith the APA
and requ ire netm on to be the d aem on that p olls objects in ord er to
w ork p rop erly. Do not enable the APA if these p lu g -ins w ere
p u rchased .
Turn APA polling on or off (“the big sw itch”)
To enable APA, follow these step s:
1. Read $OV_DOC/ w hitepapers/ Active_Problem_Analyzer.pd f or the
Using_Extend ed_Topology user m anual after V7.53
2. Exit GUI sessions
3. Run: setupExtTopo.ovpl, then etrestart.ovpl. Wait
4. Run ovet_apaConfig.ovpl -enable APAPolling
5. Watch ovstatus carefully :
ovstatus –v netmon
Should say “Polling 0 interfaces”
ovstatus –v ovet_poll Should say “Polling d evices”
6. If netmon is still polling, run: ovstop and ovstart
To d isable the APA, ru n:
ovet_apaConfig.ovpl -disable APAPolling
D etermine poller control
To u nam bigu ou sly d eterm ine w hether netmon is p olling for statu s or
the APA, ru n:
ovet_apaConfig.ovpl -query APAPolling
APA configuration in a nutshell
The follow ing xm l files are u sed to configu re the APA:
$OV_CON F/
$OV_CON F/
$OV_CON F/
$OV_CON F/
nnm et/
nnm et/
nnm et/
nnm et/
paConfig.xml
topology/ filter/ TopoFilters.xm l
topology/ filter/ APANoPollN od es.xm l
topology/ filter/ MyH ostID.xm l
The follow ing tools are u sed to tu ne APA settings:
/opt/OV/bin/ovet_topodump.ovpl
/opt/OV/support/NM/checkpollcfg
/opt/OV/bin/ovet_demandpoll.ovpl
71
Fognet’s Field Guide to OpenView NNM
D emand polling using ovet_demandpoll.ovpl
The
GUI-based
“p oll
nod e”
and
the
u nd erlying
com m and
nmdemandpoll shou ld force a netmon d iscovery p oll, bu t not a statu s
p oll for APA nod es. In N N M 7.01 and u p , ovet_demandpoll.ovpl is
available for forcing APA statu s p olls. If issu ed w ith the –d op tion,
d etailed inform ation abou t the internal states of the d evice being
p olled w ill be d u m p ed , bu t the actu al d ev ice w ill not be actively
p olled . Below are all op tions, som e of w hich are not su p p orted in
earlier releases of N N M:
-o
-d
-t
-r
-V
-v
-B
-s
-P
-p
-g
Sp ecify a sp ecific object (su p p ly OID)
Du m p the statu s of the object
st
Tim eou t to w ait for 1 resp onse
Sp ecify the Overlap p ing Dom ain ID
Verbose
Use to show H SRP virtu al IP
Force sync betw een legacy and ET d atabases
Generate su m m ary rep orts
Enable logging of p olling activity (requ ires –o)
Disable logging of p olling activity (regu ires –o)
Create sam p le island Grou p config file
The –g op tion above w as ad d ed in N N M 7.53 and is d iscu ssed in the
section below on Island Grou p Monitoring.
The –d op tion above changes after installing N N M 7.51 Interm ed iate
Patch 15, p rovid ing m ore d etailed info on u se of ET filters.
Exam p le ovet_demandpoll.ovpl –d <node> ou tp u t:
OBJ_TYPE
SHORT_OBJECT_NAME snmpEnable ET_FILTER
------------------------------------------------------------all-types
*
true
DEFAULT
NODE
nodename
true
isRouter
Composite
nodename
true
Node Value
OBJ_TYPE
SHORT_OBJECT_NAME pingEnable ET_FILTER
------------------------------------------------------------all-types
*
true
DEFAULT
NODE
nodename
true
isRouter
Composite
nodename
true
Node Value
OBJ_TYPE
SHORT_OBJECT_NAME interval
ET_FILTER
------------------------------------------------------------all-types
*
300
DEFAULT
NODE
nodename
300
isRouter
Composite
nodename
300
Node Value
72
Status Polling - APA
N ote in the above ou tp u t, the resu lts show the SN MP, Ping, and
interval settings for each interface on the nod e.
Exam p le ovet_demandpoll.ovpl –d <interface> ou tp u t:
OBJ_TYPE
SHORT_OBJECT_NAME snmpEnable ET_FILTER
------------------------------------------------------------all-types
*
true
DEFAULT
NODE
nodename
true
isRouter
INTERFACE
nodename[0[1]]
true
UnconnectedIF
Composite
nodename[0[1]]
true
InterfaceVal
If ovet_demandpoll.ovpl is issu ed w ith the –V op tion, d etailed
inform ation w ill be d u m p ed abou t the act ively issu ed p oll. If issu ed
w ith the –B op tion, the statu s brid ge w ill be force-u p d ated . Use this to
help u p d ate stale statu s that is show ing in the IPMAP top ology or if
the Dynam ic View s m ap s show inconsistent statu s w ith the ovw m ap ‟s
statu s for a p articu lar object.
On w ind ow s, the ovet_d em and p oll.ovp l scrip t calls an execu table
called $OV_SUPPORT/ N M/ ovet_d em and p oll.exe. Du e to the w ay
this is stru ctu red , ou tp u t from the scrip t can‟t be red irected to stand ard
ou tp u t. To w ork arou nd this, cop y ovet_d em and p oll.exe into the
$OV_BIN d irectory and then ru n ovet_d em and p oll.exe instead of
ovet_d em and p oll.ovp l. The argu m ents are id entical and the ou tp u t
can be red ict4ed , for exam p le to a file:
ovet_d emand poll.exe –d patchy.fognet.com > c:\ tem p\ patchy.dm p
APA Island Group Monitoring
Introd u ced in N N M 7.53, this featu re p rovid es a m ethod for hand ling
“island s of connectivity” that resu lt from ET‟s inability to d iscover
connectivity throu gh WAN clou d s. The featu re is enabled by d efau lt,
bu t configu ration of the island grou p nam es is necessary in ord er to
p rop erly id entify the island grou p nam es that N N M w ill rep ort on.
This rep orting is conveyed throu gh the follow ing tw o new APA
events:
OV_APA_ISLAND_GROUP_DOWN
OV_APA_ISLAND_GROUP_UP
The Grou p Dow n alarm m ight look like the follow ing:
Rem ote Site N od e Group Dow n 3 Capabilites: isIPRouter,isSw icth,
73
Fognet’s Field Guide to OpenView NNM
The 2 varbind s p assed after the “Grou p Dow n” text are the grou p
nu m ber and the grou p nam e, w hich by d efau lt m ean little u nless those
attribu tes are exp licitly cod ed p er the p roced u re below .
ET d iscovers nod es and their connectivity, bu t not the grou p s. The
APA bu ild s grou p s w hen load ing d ata from ET. By d efau lt, a single
nod e in the grou p , typ ically a rou ter is selected to rep resent the grou p .
A label w hich by d efau lt is blank is then available to nam e the grou p .
To cu stom ize this behavior, follow these step s:
1.
2.
3.
Run ovet_d em and poll.ovpl –g to create a sam ple configuration file
called $OVCON F/ nnm et/ paIsland GroupN am eSam ple.txt
Ed it the file to ad d group nam es and , if d esired , change the
representative nod e for the group. The d efault nam e chosen by
APA can be used if d esired . The nam e m ay contain spaces.
Restart ovet_poll using ovstop/ovstart
WAN connections w here ET can d eterm ine connectivity, like fram e
relay links w here CDP is ru nning over them , w ill not be consid ered
island s. Bu t rem ote sites that are connected throu gh MPLS clou d s w ill
be consid ered island s, since ET can‟t resolve that p articu lar sort of
connectivity. Island Grou p Monitoring is enabled (d efau lt) or d isabled
via the p aConfig.xm l file p aram eter disableIslandGroupDiscovery.
APA status events
APA-generated statu s events are d ifferent from netmon-based statu s
events, for exam p le: OV_APA_IF_DOWN (Op enView enterp rise sp ecific
event nu m ber 58983012). With APA p olling, ICMP and / or SN MP
p olls relate to ad d ress, interface, Aggregated Interface, N od e,
Connection, and Board . In the APA, p olling granu larity is d efined by
ET Top ology filters. “Unreachable” is the term u sed in the APA to
d escribe second ary entity failu re statu s.
APA status from SN MP traps
APA listens for the below SN MP trap s generated by d evices to sp eed
and ad d intellignce to the APA statu s. In m any environm ents, these
trap s are d isabled at the sou rce to red u ce SN MP chatter. Assu ring that
these trap s are tu rned on at the d evice level for critical d evices m ay
increase the sp eed and accu racy of APA statu s.
CiscoColdStart
CiscoLinkDown
CiscoLinkUp
CiscoWarmStart
74
1.3.6.1.6.3.1.1.5.1
1.3.6.1.6.3.1.1.5.3
1.3.6.1.6.3.1.1.5.4
1.3.6.1.6.3.1.1.5.2
Status Polling - APA
ColdStart
WarmStart
HSRPState
LinkDown
LinkUp
chassisChangeNotifOID
StackMIBModuleDown
StackMIBModuleUp
1.3.6.1.6.3.1.1.5.1
1.3.6.1.6.3.1.1.5.2
1.3.6.1.4.1.9.9.106.2.0.1
1.3.6.1.6.3.1.1.5.3
1.3.6.1.6.3.1.1.5.4
1.3.6.1.4.1.9.5.11.2.0.2
1.3.6.1.4.1.9.5.0.4
1.3.6.1.4.1.9.5.
Suppress/allow APA status polling
To su p p ress or allow APA p olling to interfaces/ p orts u sing Dynam ic
View s, u se the Port Ad m in Tool (p age 249) if ru nning N N M V7.53.
To su p p ress or allow p olling to any APA -p olled entity (nod es,
interfaces,
connections,
board s
and
ad d resses), u se
the
ovet_toposet.ovpl com m and . This w ill w ork only for APA nod es in
N N M 7.50 and u p :
-a
-s
-o
Option allow s APA polling for that entity
Option suppresses APA polling for that entity
Specify a particular object as the target
Use this com m and in conju nction w ith ovtopofix -G to u nm anage
d evices on the netmon sid e. N od es that have been su p p ressed w ill
eventu ally be rem oved from both the ET and legacy d atabases. In
typ ical environm ents, this takes abou t 7 d ays. For exam p le, to stop
p olling to a p articu lar interface on a nod e:
1. Confirm polling status of interfaces by running:
$OV_SUPPORT/NM/checkPollCfg –o <node>
2. To m ap interface IP to interface nam es, run:
ovet_topodump.ovpl –nodeif <node>
When the interface nam e is too long, for exam ple “H P NC7781 Gigabit
Server Ad apter,” the OID string can be used instead of the ifNam e
string w hen specifying the interface in the com m and ovet_toposet
below . To get the OID string, run:
ovet_topodump.ovpl –nodeif –detail <node>
3. Disable polling on interface hme1:
$OV_SUPPORT/NM/ovet_toposet –s –nodeif <node> -if hme1
4. Repeat step 1 to confirm polling configuration change
APA status events
Selected APA varbind s of interest:
Varbind #
Description
75
Fognet’s Field Guide to OpenView NNM
--------------------------------------------------------------------------$2
Tim estam p event occured
$3
H ostnam e of nod e that caused the event
$5
Label of the responsible interface
$6
ifAlias of the responsible interface
$8
ifInd ex of the responsible interface
$9
ifDescr of the responsible interface
$10
Responsible Level 3 add ress or port #
$11
Responsible Level 2 add ress
$12
Subnet Mask
$13
Route Distinguisher
$15
Capabilities
$16-$28
Double-object failure varbind s if connector
$29-$42
Prim ary failure varbind s if secondary
Exam p le APA Event texts:
IF Down:
Address Down:
Connection Down:
Node Down:
$5 $10 $6 Capabilities: $15
$5 $10 $6 Capabilities: $15
$5 $10 connected to $16 $18
$10 Capabilities: $15
APA status as reflected in ovw maps
APA statu s is m u ch m ore accu rate and rep resentative of the p olling
entities in Dynam ic View s than it is in brid ged statu s to the legacy
top ology. The ovet_bridge d aem on is resp onsible for m ap p ing APA
statu s to ovw m ap statu s. The m ap p ing betw een ET and the ovtop m d is
controlled by a flat file called hosts.nnm . This file shou ld not be
changed by u sers. Resolve any d escrep ancies throu gh ET d iscovery or
throu gh ovw top ology d atabase m anip u lation tools su ch as ovtopofix.
Enabling APA for the general IP environm ent can resu lt in interfaces
m arked "norm al" in ovtopmd/ ovw that shou ld be m arked "critical."
When APA attem p ts to m atch interface objects in ovtopmd w ith
interface objects in the Extend ed Top ology d atabase, interface objects
that d o not d irectly m atch w ill be m arked "norm al" in the ovtop m d
d atabase and ovw, and a m essage noting this w ill be logged in
$OV_LOG/ ovet_p oll.log. If an interface exists in the ovtopmd/ ovw
d atabases in m u ltip le form s, only one of those interfaces w ill be
u p d ated .
If Layer-2 connectivity d iscovery is enabled in netmon, netmon w ill
attem p t to d o connectivity d iscovery on brid geMIB su p p orting
d evices, resu lting in extra interfaces w ith the sam e ifN u m ber u nd er
som e cond itions. APA w ill only u p d ate one of these interfaces w ith
statu s. Ad d itionally, these interfaces are som etim es d eleted and
recreated , resu lting in an inconsistency w ith the Extend ed Top ology
76
Status Polling - APA
d atabase u ntil the next ET d iscovery. Sim ilarly, som e interfaces are
created to connect sw itches or other connectors to segm ents in the
ovtop m d d atabase. These interfaces, w hich have no real interface
attribu tes, w ill also be m arked norm al.
If Mu ltip le IP Ad d resses exist on the sam e interface, ovtopmd m od els
this as tw o interfaces w ith d ifferent IP ad d resses bu t the sam e
ifN u m ber. APA and Extend ed Top ology w ill treat this as a single
interface w ith m u ltip le IP ad d resses. APA w ill u p d ate only one of the
interfaces in the ovtopmd/ ovw d atabase w ith correct statu s.
In general, ET re-d iscovery w ill resolve m ost d escrep ancies betw een
the ET d b‟s view of the top ology and ovtopmd/ ovw’s.
APA aggregated port support
N ew er APA events in N N M 7.5 su p p ort AgPort by m ap p ing m u ltip le
p hysical p orts via an ET tru nk virtu al p ort. The APA p olls p hysical
interfaces, bu t not logical interfaces.
Su p p ort is for Cisco PAgP in V7.5. V7.51 ad d ed su p p ort for N ortel
MLT and SMLT. V7.51 su p p lied a w hite p ap er on the N ortel agents in:
$OV_DOC/ WhitePapers/ MLT.pd f
In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using
Extend ed Top ology u ser m anu al. The p reviou s tru nk su p p ort via the
netm on.equ ivPorts configu ration file is now called “red u nd ant
connection su p p ort.”
Fine tu ne AgPort via the follow ing ET Top ology filters:
isAggregated IF
isPartOfAggregatedIF
virtual interface
physical interface
Aggregate p ort scenario:
Suppose one physical port goes d ow n on a trunk. In this case, a
TrunkDegrad ed event issued and the trunk virtu al port status changes
to m inor in ET. Also, the physical interface changes to critical and the
Interface Dow n APA event is correlated and em bed d ed by the
ConnectorDow n correlation . AgPort status from ET is not propagated to
the IPMAP topology and can only be observed in Dynam ic View s.
AgPort statu s events:
77
Fognet’s Field Guide to OpenView NNM
OV_APA_AGGPORT_DEGRADED: the aggregate port connection betw een
tw o nod es is respond ing to polls and som e of the interfaces are d ow n.
OV_APA_AGGPORT_DISABLED: the prim ary aggregated port is not
respond ing to polls in a norm al fashion. This could be because all the
interfaces' ifAd m inStatus are Dow n| Testing.
OV_APA_AGGPORT_DOWN : the aggregate port connection betw een tw o
nod es is not respond ing to polls and all interfaces on this side of the
connection m ay be d ow n.
OV_APA_AGGPORT_UN REACH ABLE: the aggregate port conn ection
betw een tw o nod es is not respond ing to polls. The problem is d ue to
another entity.
OV_APA_AGGPORT_CON N _DOWN : the aggregate port conn ection
betw een tw o nod es is not respond ing to polls and all interfaces m ay be
d ow n on both sid es of the connection.
OV_APA_AGGPORT_REMOVED: the SN MP query returned noSuchObj.
This can occur if the port is reconfigured of if an ind ex renum bering has
occurred . It can also be a problem w ith the system ‟s SN MP agent. This event
w as ad d ed in a patch to V7.5.
APA HSRP and VRRP status support
The N N M Ad vanced Rou ting SPI license is requ ired to u se the H SRP
or VRRP ET Device agents that the APA need s to p rovid e p rotocol
related statu s. The follow ing statu s events w ere ad d ed in a p atch to
V7.5 for VRRP su p p ort:
•
•
•
•
•
•
rcVrrpTrapN ew Master
rcVrrpTrapAuthFailure
rcVrrpTrapStateTransition
vrrpTrapN ew Master
vrrpTrapAuthFailure
snTrapVrrpIfStateChange
The follow ing APA H SRP events w ere ad d ed in V7.01 and enhanced
for VRRP Protocol su p p ort in a p atch to V7.5:
•
•
•
•
•
•
•
78
OV_H SRP_N o_Active
OV_H SRP_Multiple_Active
OV_H SRP_N oStand by
OV_H SRP_Degrad ed
OV_H SRP_FailOver
OV_H SRP_Stand by_Changed
OV_H SRP_N orm al
Status Polling - APA
• OV_H SRP_Multiple_Stand by
N ew event corellations w ere ad d ed in V7.51 to su p p ort VRRP statu s. It
is im p ortant to u nd erstand the p oll trigger featu res of these corellators.
More on these corellators can be fou nd in:
$OV_DOC/ WhitePapers/ VRRP.pd f
In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using
Extend ed Top ology u ser m anu al. APA d efau lt configu ration settings
affecting H SRP statu s are as follow s:
H SRPTransientWait
GenerateNoStandbyEvent
GenerateDegradedEvent
GenerateFailoverEvent
GenerateStandbyChangedEvent
60000
true
true
true
true
H SRPTransientWait is m illisecond s to w ait for H SRP to becom e stable
after a failover. Red u cing this nu m ber increases the chance of
u nnecessary statu s events. The rem aining configu ration settings
control the generation of interm ed iate statu s events.
APA board status support
Su p p ort for board statu s is ru d im entary in N N M 7.5. Wid er su p p ort
for this featu re can be exp ected in fu tu re versions althou gh no
ad d itional su p p ort w as ad d ed in N N M 7.51 or N N M 7.53. As of V7.53,
board statu s is only su p p orted for the follow ing MIBs:
Cisco Stack MIB
Rhino MIB
C2900 MIB
Su p p ort for OLD-CISCO-CH ASSIS-MIB w as ad d ed in Intem ed iate
p atch 15 to N N M V7.51. The Board Dow n event correlation logic, as
w ell as APA event triggered p olling based on the Board Dow n trap ,
w ill not w ork w ith old er versions of the Cisco stack MIB. Any version
of the stack m ib that has m od u leTyp e in var bind 1 of the m od u le
d ow n trap , instead of m od u leInd ex, w ill not w ork. Confirm the
ap p rop riate version of the stack MIB throu gh the event configu ration
GUI.
When a Su bBoard is d iscovered , it is treated as a board . Board s w ith
u nreachable statu s im p ly a second ary failu re.
79
Fognet’s Field Guide to OpenView NNM
APA board statu s event varbind s:
Varbind #
Description
---------------------------------------------------------------------------------$2
Tim estam p event occurred
$3
N am e of the nod e that contains the board
$5
Capabilities
$6
Managem ent Ad d ress
$7
Route Distinguisher
$9
Ind ex of the responsible board
$11
SubBoard ind ex of the responsible board
$13
Serial N umber
$14
Mod ule Nam e
$15
Mod ule Description
$16
Hard ware Version
$17
Software Version
paConfig.xml: the APA configuration file
The location of this file is: $OV_CON F/ nnm et/ p aConfig.xm l.
The schem a is d efined in: p aConfigSchem a.xsd .
Changes take affect w hen ovet_poll is restarted via ovstart.
Rem em ber to backu p p xConfig.xm l file before m aking changes.
Cu stom izations need to be m erged in on u p grad es and p atches.
p aConfig.xm l cu stom ization and m od ification best p ractices are:
1. Backup the paConfig.xm l file, track revisions for reverting
2. Docum ent netmon and APA configuration custom izations
3. Use ovet_topodump.ovpl to test that the nod es or interfaces pass the
class filter created or mod ified
4. Valid ate XML syntax using an xm l ed itor or w eb brow ser
5. Test changes using checkPollCfg in support subd irectory
6. Restart ovet_poll process to begin using the new settings
7. verify w hich filter is used for node and/ or interface w ith
ovet_d emand poll -d nod enam e
Sim p lified p aConfig.xm l schem a w ith p aram eterList exam p les:
<paConfig>
<subSystemConfig>PollingEngine,StatusAnalyzer,Talker
<globalParameters>statisticsEnable
<configGroupList>
<configGroup> pollingSettings;configPollSettings
<generalParameters> GenerateDegradedEvent
<classSpecificParameters>
<defaultParameters>interval;snmpEnable;timeout
80
Status Polling - APA
<classSpecification> isRouter, isSwitch,
<parameterList> interval;snmpEnable;timeout
paConfig.xml evaluation order issues
Sp ecifications are evalu ated in top to bottom ord er , so the first
sp ecification that p assed the filter for a p articu lar d evice is the one that
ap p lies.
In N N M 7.01, an IF filter w ou ld take p reced ence over the nod e level
for statu s sp ecifications if the IF filter ap p eared before the nod e sp ec.
In N N M 7.5, the above behavior w as changed so that a statu s setting of
false for a nod e m akes that statu s false for every IF on that nod e. For
the tru e case, how ever, a false IF setting changes the statu s of the IF on
a nod e set to tru e from false.
In N N M 7.01, if a filter sp ecifies a statu s for p ing, bu t not SN MP (or
vice versa), the d efau lt p aram eter w ou ld be ap p lied to the m issing
sp ec and all other sp ecs w ou ld be ignored .
In N N M 7.5, m u ltip le filters can be evalu ated and the first filter to
sp ecify a sp ec w ill be evalu ated . Only after all sp ecs have been
evalu ated and no sp ec for either p ing or SN MP has been fou nd shou ld
the d efau lt be ap p lied .
Interface ICMP polling
Prior to N N M 7.51 Interm ed iate Patch 18, the APA w as hard cod ed to
m ark all the IP interfaces as p olled d isabled and for ICMP, and this
cou ld not be changed via the p aConfig.xm l file. That p atch introd u ced
the isDiscoContrivedIF filter and by setting the pingEnable valu e
to tru e in this filter, IP interfaces can be m onitored u sing ICMP.
Synchronizing poll settings w ith configuration polls
N N M 7.5 introd u ced the configu ration p oll, so if statu s p olling is being
d isabled to a nod e or interface in the PollingSettings config grou p , it
m ay be a good id ea to d isable p olling in the ConfigPollSettings config
grou p as w ell. Configu ration p olls are alw ays SN MP.
paConfig.xml polling granularity class specifications
Polling granu larity in the APA is d efined by classSp ecifications w hich
corresp ond to top ology filters (d iscu ssed below ) that are called w ithin
each classSp ecification block.
81
Fognet’s Field Guide to OpenView NNM
In p aConfig.xm l, these settings are located u nd er:
<subSystemConfig>
<name>PollingEngine</name>
<configGroup>
<name>PollingSettings</name>
<classSpecificParameters>
<defaultParameters>
<classSpecifications>
The follow ing class sp ecifications d efine the p olling granu larity for the
APA p oller in the p aConfig.xm l file. Details on the m eanings of these
sp ecifications can be fou nd in the APA w hite p ap er. The entries below
are for N N M 7.51. They are u nchanged from N N M 7.5 excep t for the
ad d ition of the isN ew N od e and isN ew Inter face sp ecs. Und er N N M
7.01, there w as a m u ch m ore lim ited set of sp ecifications:
Lines in italic below are com m ented in p aConfig.xm l by d efau lt and
w ill not be u sed u nless u ncom m ented .
Class Specification
snm pEnable
pingEnable
_____________________________________________________________
A PA N oPollNodes
false
false
isN ew Nod e
true
true
isN ew Interface
true
true
isIpPhone
false
false
ifsWithAnycastAdd rs
n/ a
false
isRouter
true
true
AvayaIptDevices
true
true
N otConnected Snm pSw itch
n/ a
true
isSw itch
true
false
isEnd N od e
false
true
W anIf
true
false
IfTypeFilter
n/a
false
isPartOfAggregatedIF
true
n/ a
IFInN otConnected Sw itch
n/ a
true
Unconnected Ad m inUpOrTestRouterIf
true
true
Unconnected Ad m inUpOrTestSw itchIf
false
false
Unconnected Ad m inDownRouterIf
false
false
Unconnected Ad m inDownSw itchIf
false
false
Unconnected End Nod e
false
true
N otConnectedIF
false
false
A llBoards
false
n/a
N oPingA ddresses
n/a
false
Using ET topology filters to specify polling granularity
p aConfig.xm l class sp ecifications are d efined u sing extend ed top ology
filters. Extend ed top ology filters are a sep arate entity from N N M
trad itional filters.
To see a list of all existing filters, ru n:
ovet_topodump.ovpl –lfilt
82
Status Polling - APA
To see a d u m p of d iscovered d evices that p ass a given filter, ru n:
ovet_topodump.ovpl -node -filt [filtername]
ClassSp ecification filters are evalu ated in xm l file ord er, so if a d evice
m atched both isSw itch and isRou ter, the isRou ter ru le w ou ld ap p ly.
Extend ed Top ology Filters are d efined in:
$OV_CON F/ nnm et/ topology/ filter/ TopoFilters.xm l
V7.51 p rovid es a d iscu ssion of ET Top ology Filters w ith exam p les in:
$OV_DOC/ ETFilter.pd f
In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using
Extend ed Top ology u ser m anu al.
Filter assertion type attributes
Prior to V7.53, ET Filter assertion typ e attribu tes w ere not d ocu m ented ,
bu t Kevin Sm ith of H P d ocu m ented them in his N N M 7.51
Dep loym ent hand book w hich can be fou nd at:
http:/ / w w w.fognet.com/ NN M_7.51_Deploym ent_H and book_v1.3.pd f
Assertions are d ocu m ented in the V7.53 Using_Extend ed _top ology .p d f
u ser m anu al bu t are listed here for those ru nning p reviou s versions.
Assertions for Interface Containers, H SRP Grou p s and Ad d resses w ere
d efined in Kevin‟s d ocu m ent and m ay not have been su p p orted p riot
ton V7.53. Use the assertions below to bu ild cu stom ET filters for APA
statu s or for Container View s:
N ode Assertion Attribute Types
Attribute Type
Description
---------------------------------------------------------------------------------------------------nam e
The internal nam e of the nod e
SysN am e
SysN am e of the nod e
lastUpd ateTim eUTC
The last tim e the nod e was updated
d escription
The d escription of the nod e
IPAd d ress
Ad d ress on the nod e
sysOID
Match on SN MP System OID of the node
capability
The capability of the node
status
The overall status of the nod e
extensibleAttribute
An “extensible” attribute of the nod e
H ostIDFile
Use H ostNam e or IP Ad d ress in the file
Card Assertion Attribute Types
Attribute Type
Description
83
Fognet’s Field Guide to OpenView NNM
---------------------------------------------------------------------------------------------------nam e
The nnm entity nam e of the card
lastUpd ateTim eUTC
The last tim e the card was updated
ind ex
The card ind ex
d escription
The d escription of the card
type
MIB type field on the Card
m od el
MIB mod el field on the Card
sn
MIB serial num ber field on the Card
fw version
MIB firm w are version field on the Card
hw version
MIB hard ware version field on the Card
sw version
MIB software version field on the Card
com ponentNam e
SN MP System OID of the card
status
The overall status of the card
extensibleAttribute
The extensible attributes of the card
card Adm inStatus
Card ad m instration status
cardOperStatus
Card Operation Status
m ibType
MIB type from w hich card d ata was read
Interface Assertion Attribute Types
Attribute Type
Description
---------------------------------------------------------------------------------------------------lastUpd ateTim eUTC
The last tim e the interface was upd ated .
IPAd d ress
IPv4 ad d ress bound to the Interface
ifDescription
The d escription of the interface
ifDesc
The d escription of the interface
ifAlias
Interface N am e Alias
ifN am e
Interface N am e Alias
ifInd ex
Interface Ind ex
vlanPortType
The role of this port in VLAN config
ifAd m inState
Interface ad m instration state
ifOperStatus
Interface Operation Status
ifType
Interface Type
ifSpeed
Interface Speed
status
The overall status of the interface
extensibleAttribute
The extensible attributes of the interface
capability
The capability on the Interface
Interface Container Assertion Attribute Types
Attribute Type
Description
---------------------------------------------------------------------------------------------------nam e
The nnm entity nam e of the If container
lastUpd ateTim eUTC
The last tim e the If Container was upd ated
d escription
The d escription of the If Container
type
MIB type field on the If Container
status
The overall status of the If Container
extensibleAttribute
The extensible attributes of the If Container
HSRP Group Assertion Attribute Types
Attribute Type
Description
---------------------------------------------------------------------------------------------------virtualAd d ress
Virtual Ad d ress of the H SRP Group
Address Assertion Attribute Types
Attribute Type
Description
----------------------------------------------------------------------------------------------------
84
Status Polling - APA
IPAd d ress
reachabilityState
extensibleAttribute
IP Ad d ress
state of the ad d ress
The extensible attributes of the add ress
Filter nodes based on SN MP sysObjectID
In this exam p le, nod es w hose SN MP agent m atches Blu ecoat SN MP
SysOID 1.3.6.1.4.1.3417.1.1.23 w ill be not p inged .
Ad d the follow ing nod e filter to Top oFilters.xm l:
<nod eAssertion nam e="isBlueCoat" title="isBlueCoat"
d escription="BlueCoat d evices">
<operator oper="N OOP">
<attribute>
<sysOID>1.3.6.1.4.1.3417.1.1.23</ sysOID>
</ attribu te>
</ operator>
</ nod eAssertion>
N ext ad d an interface assertion filter to m atch all interfaces on these
nod es:
<interfaceAssertion nam e="IFInBlueCoat" title="IFInBlueCoat"
d escription="Interfaces in BlueCoat d evices">
<operator oper="N OOP">
<interfaceAssociation ascType="inN od e">isBlueCoat
</ interfaceAssociation>
</ operator>
</ interfaceAssertion>
Valid ate the filters w ith ovet_topodump.ovpl then ad d the follow ing
tw o class sp ecifications near the top of p aConfig.xm l file:
<classSpecification>
<filterN am e>isBlueCoat</ filterN am e>
<param eterList>
<param eter>
<nam e>snm pEnable</ nam e>
...
<value>true</ value>
...
</ param eter>
<param eter>
<nam e>pingEnable</ nam e>
...
<value>false</ value>
...
85
Fognet’s Field Guide to OpenView NNM
</ param eter>
</ param eterList>
</ classSpecification>
<classSpecification>
<filterN am e>IFInBlueCoat</ filterN am e>
<param eterList>
<param eter>
<nam e>snm pEnable</ nam e>
...
<value>true</ value>
...
</ param eter>
<param eter>
<nam e>pingEnable</ nam e>
...
<value>false</ value>
...
</ param eter>
</ param eterList>
</ classSpecification>
Ellip ses in the above xm l rep resent sections left ou t that can be cop ied
from sim ilar d efinitions. Valid ate changes u sing checkpollcfg –o
<IP
of
node>,
then
restart
ovet_poll and
then
ru n
ovet_demandpoll.ovpl <IP of node>.
Force a device to be polled via ICMP or SN MP only
To accom p lish this, bu ild a top ology filter and a corresp ond ing entry
in p aConfig.xm l, p erhap s by IP Ad d ress. In the p aConfig.xm l file, cop y
the configu ration for a d evice that is cu rrently filtering for that nod e,
and p lace it above the existing entry in the file. Then m od ify the cop ied
configu ration and change the ICMP or SN MP p olling booleans. Test
the changes w ith checkPollCfg then restart the ovet_poll d aem on.
In ord er to force p olling via ICMP only, a qu ick and d irty alternative is
to set an invalid SN MP com m u nity string for the d evice in the SN MP
configu ration GUI. N ote that changing SN MP configu ration retries
and tim eou ts w on‟t help , thou gh, as the SN M P Configu ration p ollerrelated settings only ap p ly to netmon’s p olling intervals. SN MP tim eou t
and retry settings for APA are set in p aConfig.xm l in the
ConfigPollSettings configu ration grou p . Only the com m u nity string
and SN MP version settings are read in tot he APA from the SN MP
Configu ration GUI (xnmsnmpconf).
86
Status Polling - APA
Filtering by ifType (APA)
Tw o d efau lt filters ship w ith p aConfig.xm l in N N M 7.5 and
filters are com m ented ou t by d efau lt:
both
 IfTypeFilter sets ping to false for m atching types; useful to prevent polls
from “w aking” ISDN interfaces, etc.
 W anIf stops APA polling of m atching types in ord er to suppress
connection level APA status events
W anIf is d efined as w anIfTyp es filter and ed w ith slow IfSp eed s filter,
w here slow IfSp eed s inclu d e: 9k, 16k, 56K, 64K.
To enable either of these filters, rem ove the com m ents at the end and
the beginning of the filter d efinition; then m od ify the d efau lt ifTyp es in
Top oFilters.xm l (if d esired ), then ru n: ovstop ovet_poll and
ovstart ovet_poll.
For interfaces that are u nd esirable, netm on.interfaceN oDiscover is
u sed by netm on to p revent d iscovery and it it is also u sed by ET to
p revent d iscovery. More on p age 53.
A com p lete list of ifTyp es can be fou nd at:
w w w.iana.org/ assignm ents/ ianaiftype-m ib
D isable ICMP to a firew all (APA)
This requ ires establishing a new nod e assertion and new class
sp ecification. H ere are the step s:
1. Backup paConfig.xm l and TopoFilters.xm l
2. Determ ine the SN MP sysObjectID of the firew all:
3. N eighbor View , Right Click, Details, or run:
snmpget –T <firewallName> system.sysObjectID.0
4. In TopoFilters.xm l, copy and paste an OID-based nod e assertion block
5. Change nam e, title, d escription and OID block to m atch firewall d evice
6. Check xm l syntax and confirm filter m atches the d evices by running:
ovet_topodump.ovpl –node –filt <newNodeAssertionName>
7. In paConfig.xm l, copy entire isRouter ClassSpecification
8. Paste before isRouter ClassSpecification
9. Change the ClassSpecification nam e to m atch new nod eAssertion nam e
10. Change pingEnable param eter to false
11. Check xm l syntax and confirm polling settings:
$OV_SUPPORT/checkPollCfg –o <firewallName>
87
Fognet’s Field Guide to OpenView NNM
12. Apply changes by running ovstop/ovstart on ovet_poll
Sw itching routers and routing sw itches in the APA
If a nod e is both a sw itch and a rou ter, som etim e called a sw ou ter, then
APA p olls it as a rou ter by d efau lt. That is d u e to the fact that the
isRou ter class sp ecification is above the isSw itch class sp ecification in
the p aConfig.xm l file. By d efau lt, a sw itch/ rou ter is both SN MP p olled
and p inged . It also m eans that Connected and Unconnected Ad m in UP
interfaces w ou ld be both SN MP p olled and p inged .
With the release of N N M 7.53, the follow ing d ocu m ent inclu d es an
ap p end ix w hich w alks throu gh a m ethod ology for hand ling d evices
w ith both sw itch and rou ter fu nctionality:
$OV_DOC/ w hitepapers/ ETand APADeploym entGuid e.pd f
If ru nning versions p rior to 7.53, Kevin Sm ith of H P had w ritten the
“N N M 7.51 Dep loym ent H and book” w hich contained the exact sam e
m ethod ology and also, coincid entely p erhap s, ap p ears in Ap p end ix A
of his d ocu m ent. This hand book can be d ow nload ed from :
w w w.fognet.com / NN M_7.51_Deploym ent_Hand book_v1.3.pd f
Interesting sw itch interface filter example
The follow ing exam p le p rovid ed by N ils Johannessen d em onstrates a
typ ical “InterestingSw ou terIF” assertion that is introd u ced in the
section above. In this exam p le, the list of interesting interfaces is
lim ited to interfaces having IP ad d resses w ithin a range as w ell as
those being insid e that range and also lim ited to the set of ifTyp es
given:
!-- added 2007-09-12 Nils A Johannessen - Manag-E Nordic->
<interfaceAssertion name="InterestingSwouterIF" title=""
description="">
<operator oper="OR">
<attribute>
<capability>isL2Connected</capability>
</attribute>
<attribute>
<ifType>24</ifType>
<!-- 24 = loopback -->
</attribute>
<attribute>
<ifType>53</ifType>
<!—- 53, 135,136 = VLAN -->
</attribute>
<attribute>
<ifType>135</ifType>
88
Status Polling - APA
</attribute>
<attribute>
<ifType>136</ifType>
</attribute>
<attribute>
<ifType>131</ifType>
<!-- 131 = tunnels -->
</attribute>
<attribute>
<IPAddress>
<IPv4>
<address>10.150.*.*</address>
</IPv4>
</IPAddress>
</attribute>
</operator>
</interfaceAssertion>
APA topology events
The follow ing events are log-only by d efau lt, bu t can be set to log to
get variou s d etails abou t APA d ecision m aking. Alternatively, u se the
ovdumpevents com m and to view these log-only events:
OV_TOPOLOGY_Attr_Change_Notification
Issued w henever an ET topology object‟s attribute changes
OV_TOPOLOGY_Life_Cycle_N otification
Issued w hen an ET object is created or deleted
OV_TOPOLOGY_Topology_State_Notification
Rich d ata pertaining to the status of ET Discovery
OV_TOPOLOGY_Status_Change_N otification
Issued for every ET topology status change
APA and important node filters
Im p ortant nod es are d efined in the MyH ostID.xm l filter file to
configu re the APA to alw ays send u ncorrelated alarm s associated w ith
d evices to the alarm brow ser. While the references to this filter in the
xm l files sp eak of byp assing the sp ecific correlation that id entifies
second ary failu res, in fact this filter byp asses all ECS correlations. The
file accep ts nod e nam es or IP ad d resses and has good exam p les of
w ild card sp ecifications. The m yH ostID.xm l file is called by the
Im p ortantN od e filter d efined in the Top oFilters.xm l w hich is called in
tu rn by the Im p ortantN od es ClassSp ecification w ithin the
p aConfig.xm l file. The file is located in:
$OV_CON F/ nnm et/ topology/ filter/ MyH ostID.xm l
89
Fognet’s Field Guide to OpenView NNM
There is a know n p roblem w ith this file in N N M 7.53 and earlier w here
an error is generated w hen MyH ostID.xm l has too m any entries. H P
recom m end s u sing IP w ild card s or IP ad d ress ranges to lim it the
nu m ber of entries in this file. After u p d ating this file, stop and restart
the ovet_poll p rocess.
APA performance improvements
APA p olling statistics are available both in log-only events and
throu gh the Dynam ic View s “Polling/ Alanysis Su m m ary” tab. The
follow ing p erform ance im p rovem ent w as su ggested by H P w ith the
release of V7.53:
In p aConfig.xm l, com m ent ou t the
IFInN otConnected Sw itch
filter,
then
change
the
isSw itch
classSp ecification p ingEnable from “false” to “tru e,” and then change
the Unconnected Ad m inUp OrTestSw itchIF classSp ecification p ing
setting from “false” to “tru e.” Test the new settings w ith checkp ollcg
and restart the ovet_poll d aem on.
Another p erform ance im p rovem ent it to increase the APA thread
cou nts by increasing the follow ing p aram eters in p aConfig.xm l:
PollingEngineThread PoolSize
statusAnalyzerThread PoolSize
Connected vs unconnected interface APA status
Prior to N N M V7.50, the APA calcu lated nod e statu s based on the
statu s of all the p olled interfaces w hether connected or not connected
in ET. In V7.51 this changed so the APA w ill not p rop agate the failu re
of u nconnected interfaces to the nod e‟s statu s.
The below p aConfig.xm l flag can be u sed to fine tu ne this new
algorithm . If the valu e of this flag is tru e APA w ill p rop agate the statu s
of u nconnected interface to the nod e and vice versa. By d efau lt the
valu e of this flag is tru e for all the interfaces excep t for all the
u nconnected Ethernet interfaces in connected nod es:
propagateStatusForUnconnectedObj
To revert back to N N M7.50 behavior, search for the follow ing
p aConfig.xm l filter and set the flag to tru e. N ote that this m ay cau se
container view to show incorrect statu s of netw ork :
UnconnectedEthIFInConnectedNode
90
Status Polling - APA
APA memory footprint reduction
N N M v7.51 introd u ced this enhancem ent w hich only load s actively p olled interfaces into m em ory. By d efau lt, the APA load s all
d iscovered interfaces into m em ory. To im p lem ent this im p rovem ent,
locate the below p aram eter in the p aConfig.xm l and change its
varValu e from false to tru e, then restart ovet_poll:
load OnlyPolled ObjectsIntoMem ory
N ote that in the V7.53 d ocu m ent listed below , sp ecial consid erations
need to be taken into conseration on N N M system s ru nning H P-UX
w hen im p lem enting the m em ory footp rint red u ction:
$OV_DOC/ w hitepapers/ ETand APADeploym entGuid e.pd f
D isabling status bridge
Those w ho rely soley on H om ebase view s and are u sing APA can
im p rove p erform ance by d isabling the statu s brid ge. This m eans that
ovw-based view s w ill not have active statu s. Sim p le set the follow ing
p aConfig.xm l p aram eter to false and restart ovet_poll to accom p lish
this:
StatusBridgeEnabled
Characterizing APA polling behavior
One w ay to d iscover w hat filters are in effect for a p articu lar set of
nod es is to change the p olling interval for a filter to a u niqu e valu e .
Look for that u niqu e valu e in the ou tp u t of the checkPollCfg
com m and . For d etailed tracking of w hat the APA is d oing, enable the
APA Top ology events d escribed in the section im m ed iately above. For
exam p le, if the p olling interval for the isRou ter filter w as changed
from 300 second s to 299 second s, then those resu lts from the
checkPollCfg com m and that show ed a p olling interval of 299 second s
w ou ld ind icate the isRou ter filter is the filter that ap p lies to those
objects.
The “ -V” and “ -d” com m and line op tion s for the ovet_demandpoll
com m and p rovid e, resp ectively, internal states of APA objects and
d etailed d u m p of the active d em and p oll.
Improving SN MP-based status w ith the APA
Defau lt APA behavior is to fail an interface that r etu rns an SN MP
noSu chObject error if the SN MP agent re-ind exes or looses track of a
91
Fognet’s Field Guide to OpenView NNM
p articu lar interface. V7.51 introd u ced a p aram eter w hich forces the
APA to p oll su ch an interface via ICMP before issu ing a failu re event.
To im p lem ent this im p rovem ent, locate the below p aram eter in the
p aConfig.xm l and change its varValu e from false t o tru e, then restart
ovet_poll:
useIcmpIfSnmpNoSuchObj
APA and ManagementAddesss picking
SN MP p referred m anagem ent ad d resses are initially d efined throu gh
netmon d iscovery, bu t w hen an SN MP m anagem ent ad d ress
su bsequ ently fails to resp ond to SN MP p olls, by d efau lt, APA p icks a
new m anagem ent ad d ress. This new ad d ress becom es the ad d ress
u sed by Extend ed Top ology as w ell.
If loop back ad d resses are enabled for m ost of the netw ork d evices in
the m anaged environm ent, it is not necessary for the m anagem ent
station to p ick a new ad d ress since m ost d evices w ill re -assign the
loop back to another w orking interface if the cu rrent interface fails.
For this reason, H P recom m end s d isabling the p ickMana gem entAd d r
featu re if loop backs are w id ely u sed in the environm ent. To d o this, for
V7.01 throu gh V7.51, change the p aConfig.xm l file p aram eter
MgmtAddrInhibited from ―false‖ to ―true‖ and restart
ovet_p oll. V7.53 introd u ced a new p aConfig.xm l p aram eter called
DisablePickMgmtAddress, and setting this is set to ―false‖ by
d efau lt.
Also,
consid er
the
netmon
lrf
setting
-k
adjustNodeSnmpAddr=false w hich is d iscu ssed in m ore d etail on
p age 43.
If loop backs are not w id ely u sed , there are three op tional
p aConfig.xm l p aram eters introd u ces w ith N N M 7.51 that allow the
APA p ickManagem entAd d ress to be m ore finely tu ned :
• Mgm tAd d rPreferred – For a given ad d ress, if this param eter is true, the
ad d ress w ill be consid ered for a new m anagem ent ad d ress over an ad d ress
w here this param eter is false as a m anagem ent ad d ress if the current
m anagem ent ad d ress fails. In ad d ition, this ad d ress w ill be consid ered
even if it is currently not configured to be polled . By d efault, this param eter
is set to false.
• Mgm tAd d rInhibited – For a given add ress, if this param eter is true, the
ad d ress w ill never be used as a managem ent ad d ress if the current
m anagem ent ad d ress fails. By d efault, this param eter is set to false.
92
Status Polling - APA
• Mgm tAd d rMaxSnm pQueries – When the current managem ent ad d ress
fails to respond to SN MP, the pickMgm tAd d ress algorithm w ill engage.
This w ill cause no m ore than Mgm tAd d rMaxSnm pQueries sim ultaneous
SN MP queries to be issued d uring the search for a new managem ent
ad d ress. fails. By d efault, this param eter is set to 10.
To get even m ore control, u se DN S to tell N N M w hich ad d ress to u se to as
the p referred SN MP ad d ress. See p age 26 and sections su bsequ ent to that.
Troubleshooting APA
To d eterm ine a d evice‟s p olling settings, ru n:
$OV_SUPPORT/NM/checkPollCfg –o <object name>
To p rint a su m m ary of objects su bject to p olling, ru n:
$OV_SUPPORT/NM/checkPollCfg –l
The m ost com m on APA p olling p roblem s relate to m ap p ing of
isSw itch and isRou ter. isSw itch and isRou ter flags are assigned by
netmon d u ring d iscovery and these cap abilities are u sed by ET
top ology filters. Ru n throu gh these checks if certain d evices are not
being p rop erly p olled by the APA:
1. Check Firew alls for SN MP/ ICMP blocking.
2. Force isRouter w ith G flag; isSw itch w ith B flag in oid _to_type file (see page 7)
3. Brow se SN MP MIBs; check for cut tables that may prevent visibility to:
.1.3.6.1.2.1.1
.1.3.6.1.2.1.17.1
.1.3.6.1.2.1.4.20.1
.1.3.6.1.2.1.2.1
.1.3.6.1.2.1.31.1.1.1.1
.1.3.6.1.2.1.31.1.1.1.18
.1.3.6.1.2.1.4.1.0
.1.3.6.1.2.1.4.21
.1.3.6.1.2.1.4.22.1
.1.3.6.1.2.1.3.1.
systemTable for oid_to_type
dot1dBaseTable, isSwitch
ipAdEntTable
ifTable, isRouter/isSwitch
ifName
ifAlias
ipForwardingTable, isRouter
ipRouteTable, isRouter
ipNetToMediaTable, isSwitch
atTable
Sam p le qu ery:
snmpwalk <target> 1.3.6.1.2.1.1
93
8. Traps, Events and Alarms
This section covers som e basics abou t N N M‟s event infrastru ctu re. More
d etails abou t ind ivid u al events are covered in the follow ing section,
Interp reting Events.
What is a trap vs. an event vs. an alarm?
In short, a trap becom es an event, and then an event becom es an alarm .
“Event” is the generic term u sed in H P‟s d ocu m entation for w hat m ost
often is technically an alarm .
An SN MP trap is an u nsolicited , u nacknow led ged notification sen t
from an SN MP agent to an SN MP Manager. N N M w rap s SN MP trap s
w ithin the OV_EVEN T stack of the pmd d aem on. This ad d s attribu tes
to the SN MP trap that are not otherw ise d efined u nd er SN MP, su ch as
severity, logging behavior, and event category.
Au tom atic actions are also ad d ed -valu e attribu tes to m ake the trap and
event, and these attribu tes are ad d ed throu gh the event configu ration
GUI w hich is a front-end to the trap d .conf configu ration file.
A trap or an event p asses into pmd, and OV events com e ou t – they are
no longer SN MP trap s. ovtrapd is the d aem on resp onsible for receiving
trap s on UDP p ort 162 by d efau lt and bu ffering them for pmd.
An alarm is sim p ly a rep resentation of an OV event in the alarm
brow ser. Attribu tes that alarm s have inclu d e acknow led gem ents,
d eletions, event correlation cou nts and em bed d ed alarm relations.
When an alarm is d eleted from an N N M u ser‟s alarm brow ser, the
u nd erlying N N M event rem ains in the eventdb.
Configuring SN MP traps via trap macros
RFC 1215 d efines the TRAP-TYPE m acro and RFC 2578 d efines the
N OTIFICATION -TYPE m acro.
These m acros d efine a w ay of
su p p lying the trap d efinitions that d evices m ay u se w ithin a MIB
d efinition file. Essentially, a TRAP-TYPE m acro is an SN MPv1 trap ,
94
Traps, Events and Alarms
w hile a N OTIFICATION -TYPE
trap / notification.
m acro
d efines
an
SN MPv2
When load ing MIB files u sing the load MIB m enu op tion w ithin the
ovw GUI (or by ru nning: xnmloadmib), N N M d etects m acros and
p rom p ts the u ser for confirm ation to u p load em bed d ed trap
d efinitions if they exist.
If, w hen load ing a MIB, no su ch m essage is d isp layed for the u ser, the
MIB d oes not contain em bed d ed trap d efinitions. The trap d efinitions
can be m anu ally extracted from the MIB files w ithou t u p load ing the
MIBs. Becau se load ed MIBs consu m e m em ory, this m ay im p rove
N N M‟s scalability in environm ents w here m em ory is at a p rem iu m . To
ju st load the trap d efinitions w ithou t load ing the MIBs, ru n :
xnmloadmib -event -trapType -trapDetail 0 –load <MIBfile>
N ote that if the MIB load u tility is invoked from the com m and line to
load a MIB (xnmloadmib –load <file>), it d oes not give the u ser the
op tion of im p orting trap d efinitions into trap d .conf file. Ru nning
xnmloadmib w ith no op tions to invoke the GUI, how ever, w ill load the
trap d efinitions.
Configuring events or traps via trap definition files
$OV_CONF/C/trapd.conf is the configu ration file that hold s the trap
d efinitions. Som e vend ors p rovid e text files that hold N N M trap d .conf
com p atible d efinitions that can be u p load ed d irectly into N N M‟s event
configu ration. Often, the nam es of these files have the w ord “trap d ” in
their filenam es. The com m and s to d o this are:
xnmevents –load <filename>
xnmevents –replace <filename>
xnmevents –merge <filename>
See the m an/ ref p ages to d eterm ine w hich com m and is best for the
environm ent. The m ost com m on issu e w ith load ing trap d .confcom p atible files is file com p atibility errors d u e to the file form at. Often,
assu ring the very first line of the trap d .conf-com p atible file read s
“VERSION 3” resolves this issu e.
Configuring events or traps manually
The third m ethod to load N N M trap d efinitions is m anu ally, via the
N N M event configu ration GUI. Follow these step s to d o this:
95
Fognet’s Field Guide to OpenView NNM
1.
Look at the enterprise ID. If there is not a d efined enterprise m atching the trap
d efinition to be created , create a new enterprise ID using Ed it > Enterprises >
N ew . Som e nam e m ust be given to the new enterprise. In som e cases, and OID
Alias m ay be specified . See the trap d .conf man/ ref for m ore on that
2.
Select the new ly created enterprise then add a specific event that m atches the
specific num ber using Ed it > Events > N ew
3.
Select the “event message” tab and select the d esired logging behavior, i.e.
“Don't log or d isplay” or log to a particular event category
4.
If logging the event, d efine a severity and Event Log m essage. See page 99 for a
list of variable bind ing variables that can be passed from the trap to the log
m essage
5.
Optionally test the new event configuration by form atting an e vent using the
snmpnotify comm and or the script:
$OV_CONTRIB/NNM/sendMsg/sendMsg.ovpl
For m ore d etails on m anu ally creating events, see the m an/ ref p age for
trap d .conf, and associated control com m and s xnmtrap and xnmevent.
Alw ays m ake a backu p of the $OV_CON F/ C/ trap d .conf file w hen
m aking event cu stom izations. N ote that d irect ed its to the trap d .conf
are not su p p orted by H P, bu t su ch ed its (w hen carefu lly hand led ) can
be a p ow erfu l w ay to bu lk-m od ify w hole classes of event d efinitions.
When m aking m anu al ed its to the trap d .conf file, they w ill not be
activated in the event stack u ntil one of the follow ing com m and s is
issu ed :
xnmevents –event
xnmtrap -event
D rop SN MP traps from particular devices
This featu re ap p eared in V7.01. In V7.51, the ability to au tom atically
u nblock trap s once the flood su bsid es w as ad d ed . In Interm ed iate
Patch 18, the “-c” op tion w as ad d ed to allow a blocking tim e interval.
Also in V7.51, Som e d ocu m entation for this featu re w ith exam p les
ap p eared in:
$OV_DOC/ WhitePapers/ EventRed uction.pd f
The $OV_CON F/ ovtrap d .conf file is a list of sp ecific IP Ad d resses and
trap OIDs that the ovtrapd d aem on w ill reject SN MP trap s from . The
file su p p orts w ild card s p er the w hite p ap er, thou gh the ovtrap d .conf
m an/ ref p age says that w ild card s are not su p p orted .
96
Traps, Events and Alarms
Im p ortant: If u sing the APA p oller, certain trap s are p assed to the
p oller to im p rove statu s intelligence. Do not d rop all trap s from
d evices that are of concern w ith resp ect to statu s. See the section
below for the list of trap s the APA u ses for statu s.
If the –B, –b op tion is set (w ith or w ithou t the –r op tion), trap s w ou ld
be su p p ressed if a storm occu rs accord ing to these ru les:
With the –B op tion, If 1500 events com e in from a d evice at a rate
greater than 15 events p er second , block the d evice and d iscard those
events. After su p p ression starts, m onitor that ad d ress it for the next
1500 events. If the rate is still greater than 15 events p er second ,
d iscard the events for another interval, otherw ise u nblock the d evice
and allow the events to com e in. Use the –b op tion to change the
d efau lt valu e of 1500 events. Use the –r op tion to change the d efau lt
valu e of 15 events p er second .
The –c op tion enables configu ring a tim e p eriod in second s for the
sp ecified blocking criteria. If the nu m ber of trap s com ing from a
p articu lar d evice reaches the valu e configu red by "-b" or "-B" in less
than tim e d u ration sp ecified by "-c", the trap s w ill be blocked from the
d evice. This op tion cannot be u sed w ith "-r" op tion.
Once the ovtrap d .conf file is u p d ated , the ovtrap d d aem on m u st be
restarted u sing ovstop and ovstart. If ru nning u nd er the UN IX, the
d aem on can be issu ed a SIGIN T instead of a restart as follow s:
ps -ef|grep ovtrapd|awk '{print $2}'|xargs kill -2
Certain trap s shou ld not be d rop p ed u sing the p roced u re above if
p olling d evices u sing the APA, since they are p assed to the p oller t o
help d erterm ine d evice statu s. See p age 74 for a list of these trap s.
Automatically suppress SN MP trap storms
This featu re seem s to have ap p eared in V7.01 or a p atch to 7.01. The –
B, -b, and -r LRF op tion s to ovtrapd d isable SN MP trap s from a d evice
in the au tom atically w hen a trap storm occu rs. These op tions cau se
au tom atic entries to be created in the $OV_CON F/ ovtrap d .conf file.
The –B op tion su p p resses trap s if 1500 are received in a 15 second
p eriod . The –b and –r op tions allow the granu larity of the nu m ber and
rate of trap s to be exp licitly set for su p p ression. For m ore inform ation,
see the ovtrapd m an/ ref and see p age 5 for the LRF u p d ate p roced u re.
H ere is an exam p le ovtrap d .lrf file:
97
Fognet’s Field Guide to OpenView NNM
OVs_YES_START:pmd:-W -b 2000 -r 30:OVs_WELL_BEHAVED::
N ote that the –W is requ ired on Wind ow s bu t not requ ired on UN IX.
Generating ad-hoc SN MP traps or N N M events
snmpnotify is the facility for generating raw SN MP trap s that can be
sent to any SN MP m anager. Later versions of N N M d o not inclu d e the
snmptrap com m and , w hich is snmpnotify‘s
m ore fam iliar
p red ecessor. snmpnotify ou tp u t m ay either be an acknow led ged
SN MPv2C inform or an u nacknow led ged SN MPv1 or SN MPv2 Trap .
Another m ethod for generating events on the N N M server is to u se the
ovevent com m and . This com m and p rovid es N N M-sp ecific d ata not
native to SN MP trap s su ch as severity and alarm category. Both
ovevent and snmpnotify have sim ilar syntax an d p rovid e the “-d ”
op tion for d u m p ing ASN .1 d ecod es.
By d efau lt, both s nmpnotify and ovevent p ick u p d ata associated w ith
the sp ecified target from the SN MP configu ration d atabase to p ass to
the com m and su ch as com m u nity strings and tim eou ts. The “-d ”
op tion is u sefu l in trou bleshooting SN MP configu ration issu es.
Exam p les of the ovevent com m and can be seen the follow ing ovp l
scrip ts in $OV_CON TRIB/ N N M: p op u p Msg, ringBell, send Msg, and
setStatu s.
D ifferences betw een snmpnotify and ovevent
snmpnotify u ses SN MP as a the transp ort w her eas ovevent u ses TCP,
w hich som e m ay consid er “m ore reliable.” snmpnotify can be u sed to
generate a trap and it can be sent to any SN MP m anager. ovevent
form ats N N M events w hich can only be received by other cop ies of
N N M.
Trap s send u sing snmpnotify alw ays flow into ECS, so can be
correlated . Events com ing from ovevent are su bject to the pmd
ov_event stack settings in p m d .lrf, so their flow eithe r throu gh or
arou nd ECS can be controlled .
See the pnum stack op tion in the m an/ ref for ov_event for m ore abou t
ECS flow control and see p age 94 for m ore inform ation on the
d ifferences betw een trap s and events,
98
Traps, Events and Alarms
Event variable bindings
The Event Log Message, Pop -u p Wind ow Message, and Com m and for
Au tom atic Action field s in the Mod ify Events and Cop y Events d ialog
boxes u se sp ecial $ variables to p resent d ata th at w ere received w ith
the event. These sp ecial characters can help p rovid e form atted ou tp u t.
All nonp rintable characters are converted to their octal (\ 000)
equ ivalent for d isp lay in the event brow ser, or w hen p assed to the
op erator initiated (m anu al) actions. The tw o excep tion s are that a tab is
d isp layed as \ t in the alarm brow ser and as sp aces in p op -u p
m essages.
A new line is d isp layed as \ n in the alarm brow ser and as a new line in
p op -u p m essages. All nonp rintable characters are p assed u nconverted
to au tom atic actions execu ted by ovactiond.
Sp ecial Characters in action callbacks:
a
b
f
n
r
t
v
\
\\
000
Xhh
Alert (bell) character
Backspace
Form feed
N ew line
Carriage return
Horizontal tab
Vertical tab
Backslash
Use to separate elem ents in a pathnam e
Octal num ber, ranging from 000 to 177
Hex num ber, both hh characters m ust be 0-9a-fA-F
Variables in action callbacks:
$1
$#
$*
$-n
$+n
$>n
$>-n
$>+n
$x
$X
$@
$O
$o
$V
$r
$ar
The first sequential attribute of the event (varbind )
The num ber of attributes in the event
All attributes as: seq num , nam e (type): value strings
The nth attribute as: seq num , nam e (type): value string
The nth attribute as: nam e: value string
All attributes greater than n as value strings. $>0 = $*
All attributes greater than n as seq nam e (type): value
All variables greater than n as nam e: value strings
Date event received using local d ate representation
Tim e event received using local tim e representation
Epoch tim e (second s since Jan 1, 1970) using tim e_t
The nam e (object id entifier) of the receiv ed event
The (object id entifier) as a string of num bers
Event type (SN MPv1, SN MPv2C, CMIP, GEN ERIC)
The im plied "source" of the event in textual form . This
m ay not be the "true source" if proxied . See $R below
Sam e as $r except print the source as an IP add ress
99
Fognet’s Field Guide to OpenView NNM
$R
$aR
$c
$s
$N
$F
$U
$$
$C
$E
$O
$e
$A
$aA
$G
$S
$T
The "true source" of the event in textual form . If the
event w as forward ed , this d isplays the ad d ress
of the rem ote pmd' s machine
Sam e as $R except print the source as an IP ad d ress
The category the event belongs in
The severity of the event
The nam e (textual alias) as d efined in trapd.conf
The textual nam e of remote pm d 's m achine if this
event w as forward ed , else local machine's nam e
The NN M UUID of the event as a string of num bers
Print the $ character
The trap com m unity string
The trap enterprise as text string from trapd .conf
The trap enterprise as text string from Load ed MIBs
The trap enterprise as Object ID string of num bers
The trap agent ad d ress as d efined in the trap PDU
If the nam e server can resolve, print the nod e nam e
Sam e as $A except print the source as an IP ad d ress
The trap's generic-trap num ber
The trap's specific-trap num ber
The trap's sysUpTim e tim e-stam p. This is the rem ote
m achine's tim e in hund redths of a second between the
last re-initialization of the d evice and the generation of
the trap. For non-SN MPv1 events this value is 0
Event logging
Events are logged to the event d atabase. Prior to N N M 6.0, events w ere
logged to a flat file called trap d .log. The p referred m ethod for
accessing events ou tsid e the alarm brow ser is to u se the ovdumpevents
com m and .
Pre-N N M 6.0 behavior, how ever, can be restored by m od ifying the
pmd LRF file and ad d ing the follow ing sw itch:
-SOV_EVENT;t;l8
This logs events to $OV_LOG/ trap d .log and sets the logfile to roll at
8MB. See p age 5 for LRF u p d ate p roced u re. N ote that there is a
significant p erform ance hit w ith logging events to trap d .log.
There are three general logging m od es: IGN ORE, w hich d iscard s the
event entirely and is configu red via the “Don‟t log or d isp lay” op tion
in the GUI; LOGON LY, w hich send s the event to the event d ataba se
bu t d oes not send it to the alarm brow ser; and Alarm Categories,
w hich log the events to the u ser -cu stom izable set of categories w hich
can be configu red via Ed it->Alarm Categories in the event
configu ration GUI.
100
Traps, Events and Alarms
Som etim es, it is d esirable to log all ignored SN MP trap s for
trou bleshooting. N ote once again that m ost of these m ay be Op enView
enterp rise alarm s.
While not su p p orted or necessarily recom m end ed , globally searching
and rep lacing the IGN ORE flag w ith the LOGON LY flag can be easily
d one in Wind ow s and UN IX text ed itors. Alw ays m ake backu p cop ies
of trap d .conf before ed iting d irectly. In UN IX, u se sed:
cp trapd.conf trapd.orig
sed '/^EVENT/s/"IGNORE"/"LOGONLY"/' trapd.orig >
trapd.conf
xnmtrap -events
D umping the entire ev ent db
Use the below com m and s to d u m p the raw event d atabase. The m ain
reason for d oing this is to view events that m ay have fallen ou t of the
alarm brow ser becau se the alarm brow ser is lim ited in the nu m ber of
events it d isp lays. Also, “log-only” events can only be d isp layed u sing
these com m and s:
ovdumpevents
ovdumpevents –t
Dum p entire eventdb to std out
Dum p entire eventdb to std out,
then tail output (Ctrl-C to stop )
These com m and s w ill show a su m m ary of regu lar and correlated
events. N ote that ovdumpevents can be CPU-intensive and that the
p rocessCorrEvents tool is UN IX Only
ovdumpevents –s ―default‖ > event.log
$OV_SUPPORT/processEvents event.log event-summary
ovdumpevents –c ―default‖ > corr.log
$OV_SUPPORT/processCorrEvents corr.log corr-summary
Interpreting ovdumpevents output
Below is an exam p le of an event from ovdumpevents ou tp u t:
1162047602 1 Sat Oct 28 11:00:02 2006 cloud y.fognet.com N IF lan0 Up
Capabilities: isN od e Root Cause: cloud y.fognet.com lan0;1 17.1.0.58916866 84819
Each event in ovdumpevents ou tp u t is a single line of the form :
Tim eStam p Cat Tim e EventSrc SWSrc EventMsg ; Sev EventOID OVObjId
Tim eStam p is the Ep och Tim e that the event w as received .
101
Fognet’s Field Guide to OpenView NNM
Cat is the nu m eric category as d efined in trap d .conf. In the exam p le
above, it is “1” w hich is “LOGON LY.” The d efau lt catego ries are:
CATEGORY 0 "IGN ORE"
CATEGORY 1 "LOGON LY" "
CATEGORY 2 "Error Alarm s"
CATEGORY 3 "Threshold Alarm s"
CATEGORY 4 "Status Alarm s"
CATEGORY 5 "Configuration Alarm s"
CATEGORY 6 "Application Alert Alarm s"
CATEGORY 7 "Problem Diagnosis Alarm s"
Tim e is the hu m an-read able translation of the Ep och Tim e.
EventSrc is the nod e that p rod u ced the event. N ote that for statu s
events w hose origin is actu ally the Op enView server, the EventSrc w ill
be set to the target of the event. In the above exam p le, the EventSrc is
“clou d y,” w hich is the nod e Op enView d etected as having Interface
Lan0 com e u p , bu t the nod e that actu ally p rod u ced the event w as
“p atchy,” the m anagem ent server. Use the flag that follow s, SWSrc, to
help d eterm ine the tru e sou rce of Op enView events.
SWSrc is a single character rep resenting the softw are sou rce of the
event. If the sou rce is “-“ then the event sou rce is an SN MP trap
generated by a sou rce other than Op enView itself. In the exam p le
above, SWSrc is “N ,” w hich is netmon, so therefore the actu al sou rce of
the event w as Op enView . The p ossible valu es for SWSrc are:
C: xnmcollect
D: snm pCollect
E: xnm events
F: ovtopofix
I: ipmap
J: ovalarm srv
L: xnm load m ib
M: ovtopmd
N : netmon
P: pm d
R: ovrepld
T: ovtrapd
b: nnm_brid ge
c: xnm topoconf
a: Generic application
d : nmd em and poll
e: ECS Engine
i: ECS Circuit
l: loadhosts
m : netmon m ask change
n: xnm polling
o: ovactiond
p: ovspm d / ovpause / ovresum e
r: rem ote pm d
s: xnm snm pconf
t: xnm trap
6: IPV6 Polling Agent
-: Default
?: N one of the above
EventMsg is the event log m essage text.
Severity is a single nu m ber as follow s:
N ormal
102
1
Traps, Events and Alarms
Warning
Minor
Major
Critical
2
3
4
5
EventOID is the SN MP Object Id enfier of the event.
OvObjId is the Op enView Object ID, or 0 if not available.
D umping parts of ev ent db
Use the -l (ell) op tion to the ovdumpevents com m and to d u m p events
betw een the nu m ber of m inu tes sp ecified and the tim e the com m and is
ru n. For exam p le, to d u m p all events logged in the last d ay, ru n:
ovdumpevents -l 1440
The follow ing scrip t p rovid es a tem p late that can be cu stom ized to
select sp ecific events from the eventdb. By d efau lt, the scrip t d u m p s
events generated “tod ay” into files sep arated by severity, e.g.
severity.Critical.qry.ou t:
$OV_CON TRIB/ NN M/ event/ EventsBySeverity.ovpl
Ad hoc queries of the ev ent db
ovdwquery allow s d irect access to the d ata w arehou se via SQL. The
follow ing exam p le ou tp u ts all events received tod ay:
ovdwquery -u ovdb -password –ovdb –file a.qry
The file a.qry‟s contents m ight look like this:
select message from nnm_event_detail
where $BEGIN_TODAY <= event_timestamp
and event_timestamp < $NOW;
Du m p nod es from top ology:
echo ―select ip_hostname from nnm_nodes;‖ |ovdwquery
Trap and event forw arding
Forw ard ing raw SN MP trap s to other SN MP m anagers is p roblem atic
and u nsu p p orted by N N M. N N M events m ay be forw ard ed to other
cop ies of N N M, how ever. Ind ivid u al events can be forw ard ed throu gh
the event configu ration GUI (xnmtrap). Forw ard ing all events is
p roblem atic, bu t a scrip t to insert the p rop er forw ard ing d ata into the
record for every logged event d efined in trap d .conf is not too d ifficu lt
103
Fognet’s Field Guide to OpenView NNM
to w rite (m ake a backu p cop y of trap d .conf first). The p roblem is that
trap s su bsequ ently ad d ed via m acros in MIBs w ill not have the
forw ard ing inform ation ad d ed au tom atically. N ote that N N Mforw ard ed events u se TCP p ort 162, not UDP p ort 162, and only cop ies
of N N M, OVO or IBM‟s Tivoli N etView can receive them .
An alternative to u sing N N M‟s event forw ard ing facility is to set u p an
au tom ated action that creates an SN MP trap u sing the snmptrap or the
snmpnotify com m and . Another alternative is to u se N N M‟s SN MP
API or CSOV to bu ild an ap p lication that cop ies the event stream and
generates SN MP trap s to send to the target ap p lication. See p age 174
on u sing N N M‟s APIs or CSOV.
Many third p arty p rod u cts exist that can forw ard raw SN MP trap s or
N N M events. Tavve (w w w .tavve.com ) p rovid es a free toolkit that is a
p op u lar m ethod for bu lk-forw ard ing N N M events as SN MP trap s to
rem ote d estinations. Bytesp here‟s SN MP Trap Manager is another free
tool that can m anage SN MP trap s forw ard ing, and is available from
the follow ing u rl:
w w w.oid view.com / trap_fault_managem ent.htm l
Op enView Op erations p rod u ct has an agent-based trap receiver w hich
is cap able of receiving trap s from N N M in the ov_event form at from
the pmd d aem on d irectly. OVO SN MP trap tem p lates are then u sed to
filter the feed of all m essages from N N M to OVO.
When forw ard ing events from N N M ru nning u nd er Wind ow s, there is
a know n p roblem in that the forw ard ed events d o not have the original
event‟s AGEN T-ADDR p rop erly encod ed in the forw ard ed trap .
Search for H P Dou cm ent ID OV-EN 000876 for d etails on the w orkarou nd , w hich is basically to set the –W LRF sw itch for the ovtrapd
d aem on. More on setting LRF sw itches on p age 5.
OpenView enterprise and N N M-generated events
Events generated u nd er the Op enView enterp rise are those generated
by N N M itself. For Op enView generated events that concern a
m anaged
nod e,
su ch
as
statu s
events,
the
sou rce
of the event is alw ays u sed as the sou rce of the event in the alarm
brow ser, even thou gh the tru e sou rce is the N N M server itself. This is
carried throu gh the sp ecial varbind .1.3.6.1.4.1.11.2.17.2.2.0. Most N N M
statu s events also carry the sou rce as regu lar varbind . For netmonbased statu s events it‟s $2 and for APA-based statu s events it‟s $3.
104
Traps, Events and Alarms
N N M also u ses events for inter-p rocess com m u nication. Most of these
events are configu red to “d on‟t log or d isp lay ” (ignore), bu t they can
u sefu l for trou bleshooting N N M and for other p u rp oses. In event
th
d u m p s, the 5 attribu te is the Op enView d aem on that generated the
event. The OV_EVEN T m an/ ref show s w hat d aem ons corresp ond to
w hat cod e letter in the d u m p , w here M is ovtopmd, N is netmon, etc.
This is instru ctive in interp reting events. All external trap s shou ld
show ovtrapd (T) as the sou rce.
It is also instru ctive to note that netmon-based interface statu s events
are from netmon, bu t nod e-level events are from ovtopmd.
Und erstand ing that nod e statu s events are generated as the resu lt of a
set of top ology cond itions rather than from actu al statu s is very
im p ortant. A nod e u p event is generated “w hen all interfaces are u p ”
and d oesn‟t alw ays corresp ond to a nod e d ow n event and vice versa.
netmon p rovid es statu s for interface-level entities only. The APA, on
the other hand , p rovid es d irect statu s at fou r sep arate levels.
Alarm and icon status color defaults
Color d efau lts can be changed w ithin $APP_DEFS/ OVw . See p age 9
for m ore on $APP_DEFS.
Op erational Statu s Colors:
Blue
Green
Cyan
Yellow
Orange
Red
Unknown
N ormal/ Up
Warning
Minor/ Marginal
Major
Critical/ Dow n
Ad m inistrative Statu s Colors:
Beige/ Off White
Tan
Salm on
Dark Brow n
Unm anged
Testing
Restricted
Disabled
Meaning of an Acknow led ged alarm :
- Alarm show s as acknow led ged (Checked ) in all brow sers
- Alarm does not change status color propagation to category
Meaning of a Deleted alarm :
- Alarm show s as d eleted (removed ) in all brow sers
- Alarm changes status color propagation to category
- Und erlying event is not d eleted from eventdb
105
Fognet’s Field Guide to OpenView NNM
ov alarmsrv LRF settings
Perform ance tu ning p aram eters:
-a <num>
-d <num>
-s <secs>
Max events to hold in ovalarm srv internal cache
N um . of alarm s to d elete after max alarm s reached
H ow often (in second s) to save brow ser state info
User control p aram eters:
-Baucsd
-Ba
-Bu
-Bc
-Bs
-Bd
-BX
-BA
Default user control behavior
User can acknow led ge alarm s
User can un-acknow led ge alarm s
User can change an alarm‟s category
User can change an alarm‟s severity
User can d elete an alarm
Exclusive: User has none of these capabilities
Exclusive: User has all capabilities (-Baucsd )
Alarm brow ser settings (XN mevent app-defaults)
To m od ify alarm brow ser ap p -d efau lts settings:
UN IX:
Wind ow s:
Ed it or add the keyword in $APP_DEFS/ XN mevents.
Ed it the follow ing registry key:
MKEY_LOCAL_MACH IN E:SOFTWARE\ H ew lettPackard \ OpenView \ N etw ork N od e Manager\ xnm events
Som e of the m ore u sefu l of th e settings inclu d e:
m axEvents:
w arnOnDelete:
m axDisplayMsgs:
filterByMap:
read Only:
N um . alarm s viewable in brow ser (3500)
“false” turns off confirm ation on deletes
Max N um . pop ups to d isplay (unset)
Display events only for nod es in m ap (false)
Disable any/ all –Bauscd settings (false)
Additional actions
Ad d itional actions are available from the alarm brow ser m enu bar.
These can be extend ed and configu red .
To Configu re Actions:
Use the com m and s Actions: Add itional Actions to configure actions that are
registered through the LRF process (page 5).
To lau nch cu stom URL‟s from Actions: View s:
Ed it and add URL‟s in xnm eventsExt.conf, close alarm brow ser, run: ovstop
ovalarmsrv; ovstart ovalarmsrv, Open alarm brow ser.
106
9. Interpreting Events
SN MP agents are typ ically very qu iet by d efau lt in term s of trap
generation. Most events that are typ ically seen in the N N M alarm
brow ser are generated by N N M itself. These are the Op enView
enterp rise events. Mostly, these are N N M‟s p oller resu lts.
N ote that all p oller resu lts in the alarm brow ser s how the nod e affected
as the sou rce, bu t all p oll resu lts are in fact generated by the Op enView
server itself. This can be m islead ing as there is no w ay to know that the
actu al sou rce of the event w as the Op enView server versu s the nod e
sou rce d isp layed in the alarm brow ser.
N N M’s handling of generic and specific traps
The SN MPv1 p rotocol d efines seven Generic SN MP trap s as follow s:
0
1
2
3
4
5
6
Cold Start
Warm Start
Link Up
Link Dow n
EGP N eighbor Loss
Authentication Failure
Enterprise
The Enterp rise generic is sp ecial becau se it allow s for the u se of the
“sp ecific” trap field to accom m od ate vend or extensions via variable
bind ings (varbind s), so the Op enView Enterp rise N od e_d ow n event,
w hich is generated by N N M, is conveyed w ith the follow ing SN MP
d ata:
Enterprise:
Generic:
Specific:
Varbind s:
.1.3.6.1.4.1.11.2.17.1
6
58916865
1,2,3…n
Generic traps and origin event OID s
SN MPv2 introd u ced the concep t of an event OID, so generic trap s (like
au thentication failu res or link u p / d ow n trap s) w ou ld fall u nd er
enterp rise .1.3.6.1.6.3.1.1.5.1 in the MIB tree. The old m ethod w as for
107
Fognet’s Field Guide to OpenView NNM
vend ors to d efine p rivate versions of the generics w ithin their
enterp rise MIB su btrees. In ord er to accom m od ate the new hand ling of
generics w ith the old m ethod , N N M au tom atically p rep end s the trap ‟s
enterp rise follow ed by the generic trap nu m ber + 1 to the enterp rise
w henever an agent send s a generic trap along w ith a p rivate
enterp rise.
This w as set u p this w ay so a link d ow n from a Cisco rou ter, for
exam p le cou ld be d ifferentiated from a link d ow n from a N ortel
rou ter. After N N M v5, the event configu ration GUI lists generic trap s
u nd er the .1.3.6.1.6.3.1.1.5 enterp rise. Previou sly, link d ow n events, for
exam p le, w ou ld have been listed u nd er generic 3. So a Cisco Link
Dow n, for exam p le, w ou ld be configu red in the event configu ration
(trap .conf) w ith an id of:
.1.3.6.1.6.3.1.1.5.3.1.3.6.1.4.1.9
Taking this ap art, this is the SN MPv2 snm p Trap s enterp rise
(.1.3.6.1.6.3.1.1.5) follow ed by the generic trap nu m ber (3) and then the
Cisco enterp rise OID (.1.3.6.1.4.1.9.) The alarm brow ser is cod ed to
d isp lay any su ch an alarm as a generic link d ow n, i.e. as being from
.1.3.6.1.6.3.1.1.5.3.
This gives N N M u sers the flexibility to create cu stom event d efinitions
m ore generically for w hat w as form erly a m onolithic generic event. So,
for any d istinct SN MP agent OID, a d istinct event d efinition can be
created . For exam p le, link d ow n trap s for Catalyst 3550‟s w ou ld have
the OID:
.1.3.6.1.6.3.1.1.5.3.1.3.6.1.4.1.9.1.431.
Sim ilarly, the d efinitions can be m ad e higher u p the OID tree. For
exam p le, a cu stom event d efinition can be m ad e for link d ow ns for all
Cisco OIDs (1.3.6.1.6.3.1.1.5.3.1.3.6.1.4.1.9), or all link d ow n events from
all enterp rises (.1.3.6.1.6.3.1.1.5.3). The OID that w ou ld cover all
generic events from all enterp rises w ou ld be .1.3.6.1.6.3.1.1.5.
N ote that all generic trap s u nd er enterp rise .1.3.6.1.6.3.1.1.5 are set to
LOG-ON LY by d efau lt, so none of the trap s d escribed above w ill be
seen in the event brow ser u nless they are exp licitly set to log.
“Received event X, N o format in trapd.conf”
This d efau lt event is generated by H P Op enView and it rep resents an
SN MP trap that has been received for w hich the N N M event system
108
Interpreting Events
has no translation d efined in N N M. It is cap tu red by the d efau lt
enterp rise OID .1.3.6.1.4.1.
Events that are cau ght by this trap d efinition need to have event
configu rations ad d ed for them u sing one of the three m ethod s
d escribed at the begin ning of this section . Bu t before sim p ly d efining
an enterp rise to accom m od ate it, it is im p ortant to d eterm ine if the trap
is m eaningfu l and d esirable in the first p lace.
First, d eterm ine the trap sou rce. Look at the sou rce‟s nam e or IP. Is it
in the N N M Top ology (it d oesn‟t have to be)? Is the sou rce an SN MP
agent know n to N N M by SN MP OID in oid_to_type (see p age 7)? Is the
MIB for the trap ‟s enterp rise load ed ? Brow se the MIB u sing the MIB
Brow ser or u se the snmpwalk com m and .
Try to control SN MP trap s from the agents d irectly first. Every d evice
m anu factu rer p rovid es a d ifferent m ethod ology for configu ring the
SN MP agent and enabling and d isabling SN MP trap s (no stand ard
configu ration interface or conventions exist for SN MP agent
ad m inistration or cu stom ization ). Typ ically, these p aram eters are set
throu gh the d evice‟s IOS.
Lots of p op u lar SN MP MIBs can be fou n d in th e
$OV_SN MP_MIBS d irectory, and som e of these hold trap m acros. To
d eterm ine if a p articu lar MIB hold s trap m acros, look (grep ) for the
keyw ord “N OTIFICATION -TYPE” insid e the MIB file. If that fails, tr y
the vend or‟s w eb site, a general w eb search or a p u blic MIB rep ository.
Page 141 lists several good MIB rep ositories.
All p rivate enterp rise SN MP agent‟s OIDs are of the form :
.1.3.6.1.4.1.number...
N umber is the IAN A-assigned vend or ID. Any d otted nu m bers beyond
the nu m ber im m ed iately follow ing .1.3.6.1.4.1 are vend or -sp ecific
d esignations. The m aster list of enterp rises‟ nu m ber assignm ents can
be fou nd at:
w w w.iana.org/ assignm ents/ enterprise-num bers
N ote that the vend or ID m ay often be m islead ing d u e to the m ergers
and acqu isitions the high tech ind u stry is su bject to. Cisco is 9, H P is
11, Bay N etw orks is 18, etc.
109
Fognet’s Field Guide to OpenView NNM
If there are lots of Cicso OID trap s com ing throu gh w ith no form at, i.e.,
the OID starts w ith 1.3.6.1.4.1.9, then see the p roced u re for form atting
trap s on p age 108. It m ay be easiest to log into cisco.com 's CCO, and
search for the N N M Integration Utility and install it. This m ay not only
solve this p roblem , bu t m ight also p rovid e cu stom sym bols for Cisco
d evices as w ell.
Finally, it m ay be d esirable to not form at u nform atted trap s. One
reason m ay be that it‟s too tim e consu m ing to track d ow n or bu ild the
trap d efinitions. Another reason m ay be that the u nform atted trap s
convey enou gh inform ation in their varbind s that a form atted trap
d efinition isn‟t necessary.
In highly-scaled environm ents, trap d efinitions consu m e m em ory and
CPU and it m ay even becom e d esirable to d elete existing trap
d efinitions, forcing them into the enterp rise d efau lt. In either case, if
“no form at” events are frequ ent and exp ected , change the event
m essage to sim p le p ass the event d ata (for exam p le, $*) so the events
d on‟t convey that there is som ething w rong w ith N N M‟s
configu ration.
Agent in distress; “spinning in ifTable or ipAddrTable”
This hap p ens w hen an SN MP agent rep orts bad or corru p t
inform ation. Generally the agent vend or m ay need to be contacted for
ad vice. “Sp inning in ifTable” m eans that the SN MP agent has rep orted
few er interfaces p resent in the variable ifN u m ber than p resent in the
rou ter tables.
This is a know n p roblem w ith som e Cisco PIX rou ters that w as fixed
by new er cod e. If this is hap p ening, it cou ld cau se p erform ance
p roblem s on the d evice the agent is ru nning on and shou ld be
im m ed iately ad d ressed . In other w ord s, the agent really is in d istress
and m ay be ru nning loop s that consu m e CPU and / or m em ory.
“Sp inning in ip Ad d rTable” m eans that the SN MP agent is rep orting
the sam e IP Ad d ress for the sam e interface (ifInd ex) m ore than once in
the table. There m ay be other reasons these alerts are being generated ,
bu t alm ost alw ays, it is becau se there is a p roblem o n the target agent,
not w ith N N M.
OV_PhysAddr_Mismatch
This event is generated by H P Op enView . One com m on reason to see
this event is d u e to p roxy-ARP from the rou ters and / or th e d efau lt
110
Interpreting Events
rou ting config on end -nod es, w hich N N M has trou ble w ith. ProxyARP is an alternative to d efining large rou ting tables on ed ge d evices,
and m ore can be learned abou t it by read ing RFC 1027. In the case of a
host w ith a d efau lt rou te, its ARP cache has the rou ter's MAC ad d ress
as the MAC ad d ress for every non -local host it has com m u nicated
w ith.
The ARP-cache of the rou ter to w hich the non -local hosts are
connected w ill have the ap p rop riate real entries, as w ill the actu al
hosts. When OV sees the ap p arent d iscrep ancy, it generates this event.
If Proxy-ARP is in u se, it is com p letely reasonable to change this event
from an error alarm to log-only or to IGN ORE. The sam e can be said
for the OV_ARPChgN ew PhysAd d r event.
Inconsistent subnet mask
Most com m only, the d evice‟s configu red m ask is actu ally incorrect.
Som etim es, the first d evice d iscovered in a new netw ork is an
im p rop erly-configu red netm ask, so su bsequ ently-d iscovered d evices
w ith p rop er netm asks generate a flood of these events. This event is
also generated becau se N N M ap p lies a classfu l m ask becau se it d oesn‟t
have enou gh inform ation to p rop erly d eterm ine the correct m ask. In
this case, N N M is the cu lp rit.
For exam p le, su p p ose the first d evice d iscovered in the entire N N M
top ology that has a 10 ad d ress is 10.1.142.5. netmon asks the d evice for
its m ask w ith ICMP m ask requ est, bu t p erhap s the d evice d oesn't
su p p ort that or filters that qu ery ou t. netmon then qu eries the d evice
via SN MP bu t the d evice d oesn't answ er that either.
In this case, N N M creates a netw ork object called 10.0.0.0/ 8, based on
the classfu lness of the 10 netw ork, and p laces the new nod e in this new
netw ork in the top ology. netmon su bsequ ently d iscovers 10.1.142.10
and asks it for the netw ork m ask w hich answ ers 255.255.255.0, w hich
N N M then calls inconsistent becau se it believes it shou ld be 255.0.0.0.
If a nam e for the corresp ond ing su bnet nu m ber isn't fou nd in
/ etc/ netw orks, then N N M nam es the netw ork sym bol as the su bnet
nu m ber. N N M rem oves trailing zeros from the nam e.
Therefore, 11/ 8, 11.0/ 16, 11.0.0/ 24 and 11.0.0.0/ 30 m ight all look like
"11" w hen nam ed p er this m ethod . If nod es exist in both the 11.0 and
11.0.0 netw orks, inconsistent su bnet m ask events w ou ld then be
generated for the nod es in the netw ork that w ere d iscovered later; after
the m ask w as set for the “11” netw ork.
111
Fognet’s Field Guide to OpenView NNM
The solu tion is sim p ly to ad d / etc/ netm ask entries, d elete, than
red iscover the nod es. The netm asks file on Wind ow s resid es in the OS
d irectory u nd er system 32\ d rivers\ etc. Finally, loadhosts m ay be
u sed to force the m asks for netw orks to be created w hen they d on‟t yet
exist in the top ology. DN S can also be u sed to configu re netw orks w ith
corresp ond ing netnam es and netm asks, w hich gives better control
over the netm asks.
Another com m on reason this event is generated is w hen u sing
variable-length su bnet m asks (VLSM), w hich N N M norm ally hand les
fine. N N M su p p ort for VLSM w as ad d ed p rior to N N M 6.0. As
d iscu ssed above, it's p ossible to have m ore than one su bnet m ask for a
class A netw ork. For exam p le, if one u sed a 16 bit m ask (e.g., 10.1/ 16)
and another u sed a 24 bit m ask (e.g., 10.2.1/ 24). This is OK - how ever,
it w ou ld be "illegal" or w rong to have an interface w ith a 24 bit m ask
that fell in the 10.1.x.x p attern, becau se it w ou ld be inconsistent
accord ing to N N M.
To fix this, valid ate the correct m ask for the netw ork and check if it is
consistent w ith w hat N N M rep orts by right-clicking on the netw ork
icon and d isp laying the object p rop erties; the IP Map attribu tes show
the netm ask that N N M has assigned u p on d iscovery.
If this is incorrect, ovtopofix –r 10 can som etim es fix the p roblem . If
not, shu td ow n netmon u sing ovstop netmon, d elete the netw ork
sym bol from all su bm ap s, and from all d efined m ap s, then red iscover
the core d evice in the netw ork u sing loadhosts w ith the –m <m ask>
op tion (p age 42). Restart netmon, confirm the p rop er netm ask is
assigned to the netw ork sym bol, and issu e a d em and p oll to the
d evice.
In extranet environm en ts, the sam e sort of thing can hap p en to CIDRbased su bnets. If rnetstat -I on the offend ing d evice retu rns
d ifferent netm asks for itself and its rou te, the netw ork settings cou ld
be CIDR com p liant bu t not RFC 950 com p liant.
In general, N N M is very strict on RFC 950 com p liance and if there are
nod es w ith interfaces that have su bnet m asks "less" than th e class of
netw ork in qu estion , then N N M w ill generate inconsistent su bnet
m ask events.
For exam p le, rnetstat –r <nod e> ou tp u t show s:
112
Interpreting Events
Interface IP address
Network Mask
Network Address
Loopback 10.177.104.9
255.255.255.252 10.177.104.8
BRI0
10.177.104.154 255.255.255.248 10.177.104.152
To fix it, d elete the incorrect interface from N N M and re-ad d it u sing
the p rop er su bnet m ask. Use loadhosts (see p age 45) or m anu ally
ad d the interface u sing m ap op erations. Rem em ber that CIDR is
Classless Inter-Dom ain Routing, not classless A ddressing.
RFC 1518 says that it sp ecifically d oes not ad d ress "p roced u res for
assigning host IP ad d resses." It is a m istake to take rou ting p olicies and
concep ts and try to ap p ly them to interfaces. If there is not an entry in
the netm asks file or if the netm ask is not sp ecified w hen nod es are
ad d ed u sing loadhosts, N N M u ses the d efau lt netm ask for the class
of netw ork, and inconsistent su bnet m ask m essages m ay resu lt.
OV_D uplicate_IP_Address
There are several reasons these m ay be generated other than a m isconfigu ration in the m anaged environm ent. The m ost com m on reason
is the p resense of H SRP or VRRP d evices w hich have m igrating IP
ad d resses. The m ost recent versions of N N M d etect H SRP interfaces
and treat them ap p rop riately via ET and the APA, bu t netmon m ay still
com p lain. To stop this com p laining, these interfaces can be ad d ed to
the netm on.noDiscover file and then d eleted from the top ology .
In m any cases, backu p d u p licate interfaces are flagged as
ad m inistratively d ow n in the MIB tables, so som etim es m anaging
these via SN MP m ay w ork if p olling is via netmon. If d esired , the
OV_Du p licate_IP_Ad d ress event can be su p p ressed for this class of
d evice u sing the “-I” LRF sw itch for netmon (see p age 5 for u p d ating
LRFs).
When an IP ad d ress is m oved from one d evice t o another, N N M m ay
generate OV_Du p licate_IP_Ad d ress events if the m ove w asn‟t
“com p lete.” In m ost cases, d eleting the objects and red iscovering them
solves the p roblem u sing the p roced u re on p age 195. It m ay not solve
the p roblem , thou gh, if for exam p le DN S w asn‟t p rop erly u p d ated , or
if the interface w as set to ad m inistratively d ow n instead of being
rem oved from the sou rce d evice.
OV_D uplicateIfAlias
Du p licate ifAlias m essages m ay ap p ear after an u p grad e to N N M 6.31
or above. N N M u ses ifAlias by d efau lt for nam ing interfaces becau se
they are a m ore stable w ay of tracking interfaces than u sing ifN am e
113
Fognet’s Field Guide to OpenView NNM
(becau se ifN am e m ay change d u e to ind ex rem ap p ing by the SN MP
Agent). Som e agents, how ever, au tom atically set the ifAlias to
“testing” if the ifAd m inStatu s is set to “testing.”
To elim inate these m essages, configu re the offend ing interfaces w ith
d ifferent ifAliases if it‟s not too m u ch trou ble. Som e d evices‟ SN MP
agents, how ever, m ay not p erm it a change in the ifAlias. To force
N N M to revert to the legacy behavior of u sing ifN am e, set the
follow ing netmon LRF sw itch u sing the p roced u re on p age 5:
-k useIfAlias=false
Another ap p roach is to m od ify the OV_Du p licateAlias event and set
its logging behavior to “Ignore.”
If u sing the APA p oller, the p aConfig.xm l file p rovid es m u ch m ore
sop histicated m ethod s for hand ling ifAliases.
Observe that som e p hysical interfaces show u p in the interface tables
as several interfaces, and on the d evice it is only p ossible to configu re
the SN MP ifDescr on one of them . In this case, the OV_Du p licatIfAlias
alarm m ay be generated since the ifAlias m ay be id entical on the
p rim ary interface and su b-interfaces. This behavior is com m on w ith
ISDN BRI interfaces on Cisco d evices. You m ay w ant to u se
netm on.interfaceN oDiscover for su ch su binterfaces.
Authentication failures
These generic SN MP trap s are generated by SN MP agents in resp onse
to a failu re to p rop erly au thenticate an incom ing SN MP requ est. For
exam p le, an SN MP get requ est w ith the im p rop er com m u nity string.
Su ch a requ est cou ld very likely be generated by agents that are being
contacted by the N N M server itself, since in general, it issu es a lot of
SN MP requ ests.
Therefore, these trap s typ ically ind icate the need to set the p rop er
SN MP com m u nity string in N N M‟s SN MP Configu ration GUI. To
d eterm ine the string N N M is u sin g to contact an SN MP agent, ru n:
xnmsnmpconf -resolve <target>
Most SN MP agents d o not p rovid e a facility to id entify the sou rce of
the failed SN MP requ ests. Determ ining the sou rce of bad SN MP
requ ests can be accom p lished u sing a sniffer (p age 172) to p u ll the
114
Interpreting Events
sou rce from SN MP PDU of the offend ing requ ests. Detailed pmd
tracing (p age 131) w ill d u m p SN MP PDU d ata as w ell.
A few d evices d o p rovid e this d ata in p rop rietary trap s, how ever, and
N N M‟s syslog facility m ay also be of u se in this case. Rem em ber, it ‟s
som etim es hard to exp licitly d eterm ine the cau se of an Au thFail trap
becau se it is a rather lim ited resp onse from an agent nod e to a nother
nod e‟s attem p t to access its SN MP agent. To toggle SN MP
au thentication failu res off at the sou rce, u se the SN MP agent vend or‟s
im p lem entation-sp ecific m ethod s. Or, if the “w rite” com m u nity string
is know n for the agent, u se:
snmpset <node> snmp.snmpEnableAuthenTraps.0 integer 2
By d efau lt, the Au thentication Failu re event in N N M is set to “log only.” Consid er logging them as a secu rity or scalability p recau tion. At
the very least, the event shou ld be set to log a statu s on occasion to
valid ate that Au thFail trap s aren‟t cau sing p otential p erform ance
issu es.
Ru n ovdumpevents to view log-only events. It m ay not be a bad id ea to
set u p a sched u led scrip t that grep s Au thFails fr om the event log and
rep orts su m m ary find ings. In larger environm ents w here certain
volu m es of Au thFail trap s are exp ected , set the logging behavior to
“IGN ORE” to im p rove p erform ance of the N N M server .
In the trap d escrip tion for the Au thFail trap , there is a note that claim s:
“H P-UX only: On H P-UX agents, the valid com m unity nam es can be found in the
/ etc/ snm pd.conf file. The IP add ress that caused the authentication failure can be
d eterm ined by either looking in the log file (snm pd .log), or by d oing a snm pwalk on
the SN MP variable:
.iso.org.dod.internet.private.enterprises.hp.nm .snm p.
authfail.authFailTable.authFailEntry”
N either the snm p d .log nor the stated MIB variable is ever u p d ated
after bad requ ests to H P-UX instances of N N M. Still, the trap
d escrip tion continu es to claim this to be so (erroneou sly).
Unknow n or unrecognized ASN .1 type # messages
These m ay show u p em bed d ed in SN MP trap varbind field s or in the
p m d .log0 file and m ay cau se pmd and / or other d aem ons to crash. The
cau se of these errors is u su ally m alform ed SN MP trap s. pmd stack
tracing (see p age 131) m ay be requ ired to d eterm ine the trap sou rce.
115
Fognet’s Field Guide to OpenView NNM
One solu tion is to d isable the transm ission of these p articu lar trap s
from the sou rce and / or to seek u p d ated agent cod e from the vend or of
the trap sou rce equ ip m ent. It m ay also be p ossible to form at a trap
d efinition (see p age 95) that d oesn‟t u se the offend ing varbind s.
Eliminating events from undesirable systems
Regard less of w hether a nod e is in N N M top ology or not, SN MP trap s
m ay be received from d evices that aren‟t “m anaged .” This is becau se
there agents are configu red w ith the N N M server as a trap d estination .
Often this can be reset at the agent. On system s that su p p ort p op u lar
op erating system s, this is u su ally set in the snm p d .conf file. Stop p ing
u nd esirable trap s at the sou rce is id eal.
If ru nning V7.01 or above, the ovtrap d .conf file can be u sed to
su p p ress trap s from p articu lar d evices. See p age 96 for m ore
inform ation on ovtrap d .conf. If ru nning an old er version, there are
several consid erations offered below .
Som etim es a d evice is generating u nd esirable trap s for a sp ecific
enterp rise. Within N N M, a trap d efinition for that p articu lar enterp rise
can be d efined in the event configu ration and then configu red to
“Ignore” all events.
To force the N N M server to reject all trap s from all enterp rises for a
p articu lar nod e, assign an incorrect com m u nity string to the target
nod e in the SN MP Configu ration GUI. Sim ilarly, a n SN MP p roxy
cou ld be configu red w ith a bad com m u nity string, for exam p le
127.0.0.1, so the N N M server no longer issu es requ ests to it. These
tricks w ill cau se ad d itional tim eou ts and retries and m ay affect server
p erform ance. Again, the m ost d esirable ap p roach is to ad d ress each
offend ing nod e ind ivid u ally by configu ring its agent not to p rod u ce
the trap s in the first p lace.
116
10. Event Correlation
The ECS ru ntim e engine w as ad d ed to N N M in version 6.0
to su p p ort the su p p ression of cascad e failu re events. ECS circu its w ere
su p p lied by H P, and the Correlation Manager w as p rovid ed
as a front-end to ad ju st basic settings and to tu rn correlations on or off.
Until N N M 6.31, there w as no facility in N N M for enhancing ECS
correlation logic other than p u rchasing the exp ensive ECS Designer
p rod u ct.
The Correlation Com p oser , how ever, w as ad d ed in N N M 6.31 and it
p rovid es an interface for creating ad vanced logic sets that are analogou s
to ECS circu its, bu t it d oes not p rovid e the fu ll circu it d evelop m ent
cap abilities of the Designer. The Com p oser is actu ally a front-end
interface to a “su p er-circu it” in term s of ECS ru ntim e and the ECS
Designer. The com p oser is actu ally im p lem ented as a regu lar ECS circu it.
What is event correlation?
Event correlation is u sed as a generic term to d escribe the ability to
p reem p tively act on a stream of events in ord er to ad d intelligence to
and control over that stream .
One im p lication of event correlation in this context is that p ast events
are related to cu rrent events in ord er to red u ce the overall am ou nt of
events generated . To accom p lish this, event correlation tools m u st be
“state aw are,” that is, they keep track of events over tim e.
Many m od ern event correlation p rod u cts d ep loy finite d ifference
engines internally to m aintain state, and a set of logical constru ct ions
can be m anip u lated to form sp ecific logic sets (circu its) that then act on
“stream s” of m essages, interru p ting or re-d irecting their flow throu gh
the event m anagem ent system at variou s stages w ith great efficiency
and scalability.
When d iscu ssing event correlation in term s of N N M, it refers to a set of
su bsystem s w hich are heavily integrated into the N N M p rod u ct. These
117
Fognet’s Field Guide to OpenView NNM
su bsystem s inclu d e the ECS ru ntim e engine, the correlation m anager,
the correlation com p oser, variou s configu ration entry p oints w ithin the
N N M GUI and variou s configu ration files.
Correlation Manager
This is the legacy interface to event correlations bu ilt in to N N M
starting w ith version 6.01. It p rovid es a JAVA-based GUI for changing
p aram eters associated w ith N N M‟s bu ilt-in ECS circu its. Access the
Correlation Manager throu gh the N N M m enu s:
Options -> Event Configuration -> Ed it -> Event Correlation
Or, the m anager can be invoked by URL:
http :/ / host.fqd n/ OvDocs/ C/ ecs/ ecscm g.htm l (Window s)
http:/ / host.fqd n:3443/ OvDocs/ C/ ecs/ ecscm g.htm l (UNIX)
On UN IX installations, the p ort nu m ber (:3443) m u st be ap p end ed to
the N N M Server hostnam e in the URL above.
Correlation Composer
The Correlation Com p oser is accessed via the Correlation M anager.
The Com p oser can be lau nched by selecting “m od ify” after
highlighting the “Com p oser” circu it in the Correlation Manager.
Alternatively, in UN IX, the Correlation Com p oser can be started w ith
the ovcomposer com m and . In Wind ow s, the com p oser can be started
w ith:
ovcomposer –m o
ovcomposer –m d
operator mod e (read -only)
d eveloper mod e
The Com p oser is a front-end interface to a “su p er circu it,” w hich is to
say that it is a single ECS circu it that can be u sed to create a fairly w id e
range of correlation logics sp ecifically for N N M.
It p rovid es a w ay to ad d correlations for sp ecific SN MP events, and is
bu nd led w ith N N M 6.31 and higher. In essence, the Com p oser is a
fancy interface to create and m anip u late very fancy external ECS fact
stores.
The Com p oser p rovid es a few correlations that serve as exam p les as to
the sorts of logic that can be created w ith this GUI.
118
Event Correlation
The p ath to the correlator store is $OV_CON F/ ecs/ CIB and the N N M p rovid ed correlators w ith V7.5 inclu d e:
N N MBasic
OV_Connector_Interm ittentStatus
OV_Chassis_Cisco
OV_Multiple_Reboots
N odeIf
OV_N od eIf_Nod eDow n
OV_N od eIf_Prim aryIFUnknow n
OV_N od eIf_Nod eN otCorrelator
Poller
OV_Connector_Interm ittentStatus
PollerPlus
OV_Link_Interm ittent
OV_Conn_Interm ittentStatus
OV_Ad d r_Interm ittentStatus
OV_Interface_Interm ittentStatus
OV_N od e_Interm ittentStatus
For creating cu stom correlations, the Com p oser p rovid es the follow ing
correlator tem p lates:
Enhance:
Trigger the creation of a new output alarm
Mod ify the inform ation for an existing alarm
Multi-Source:
Discard , mod ify, consolidate or create new output alarm (s) based on a
set of m ultiple input alarm s
Rate:
Count like input alarm s and em it all or none plus a new alarm
ind icating the threshold was breached
Repeated :
Mod e 1: Discard d uplicates in w ind ow of first alarm
Mod e 2: Replace/ embed latest alarm in brow ser
Sam e logic as DeDup correlation w ith more control
Suppress:
Discard alarm s matching cond itions
Transient:
Monitor rate and/ or consolid ate status-oriented alarm s
Sam e logic as PairWise correlation w ith more control
119
Fognet’s Field Guide to OpenView NNM
User-Defined :
Alarm s m eeting ad vanced filter or sim ple alarm signatures invoke an
input function such as a C or Perl script.
ECS D esigner
The Designer p rovid es a fu ll and robu st interface for ECS circu it
d evelop m ent. The correlations p rovid ed w ith N N M w ere them selves
d evelop ed w ith this tool (excep t for the Com p oser). With the
introd u ction of the Com p oser, the ECS Designer is longer need ed to
d evelop m ost correlations that m irror the form and fu nction of the
existing correlations. For m ore ad vanced correlations, how ever, the
ECS Designer is a m u st.
The Designer p rovid es the interface for com p iling ECS raw circu it files
(.ecs files) into com p iled circu its (.eco files) that can then be load ed into
the ECS ru ntim e engine.
The ECS Designer is not requ ired to load com p iled ECS circu its (.eco
files) into the ru ntim e engine. This im p lies that circu its d evelop ed by
som eone else can be shared (or, as is som etim es the case , sold ) to be
load ed into an environm ent w here the ECS Designer is not installed .
Instances and streams
Insid e the ECS ru ntim e engine, events are sep arated into stream s, or
flow s of events. All of the d efau lt ECS circu its and Correlation
Com p oser instances are configu red to ru n on the d efau lt stream . It is
very u nu su al to u se m u ltip le stream s in m ost environm ents.
Even w hen events are com ing from d ifferent sou rces, for exam p le
via pmd throu gh N N M, and via opcmsg throu gh OVO, the events all
flow throu gh the d efau lt stream . Only in this w ay can events from
d ifferent sou rces su ch as this be related to each other. Only w hen
events com e from sou rces that m u st be com p letely sep arated shou ld
sep arate stream s be em p loyed .
An ECS instance is like a cop y of the ru ntim e ECS engine that is
d ed icated to a p articu lar set of event sou rces. Mu ltip le instances of
ECS ru ntim e can and d o ru n on single server and each are sep arate
in their inp u t and ou tp u t. Circu its op erating in sep arate instances d o
not interact w ith each other even thou gh they m ay all be configu red
to u se the sam e stream at the sam e tim e.
120
Event Correlation
The d efau lt instance is “1” and it is u sed for N N M‟s em bed d ed ECS
ru ntim e. It gets its feed from pmd. ecsmgr com m and s are d irected to
the d efau lt instance u nless the sp ecific instance is sp ecified w ith the
“–i” op tion. So, for exam p le, the follow ing tw o com m and s are
equ ivalent:
escmgr –i 1 –info
escmgr –info
Tw o other instances that m ay be of interest are instance 11 and
instance 12. Instance 11 is reserved for the OVO m anagem ent server
and instance 12 is reserved for the OVO agent. A UN IX OVO
m anagem ent server, then, m ay have three op erative instances of ECS
ru ntim e, bu t they w on‟t be op erative by d efau lt, since there are no ou t of-box correlations for OVO server or OVO agent.
Special correlations
The DeDu p lication correlator (DeDu p ) is sp ecial becau se it is a p ostp rocessing correlator. This is requ ired in ord er to interact w ith the
alarm brow ser in su ch a w ay as to “m ove” existing alarm s from their
tim e-based p osition in brow ser. To accom p lish this, the correlation
externally com m u nicates d irectly w ith the ovalarmsrv p rocess. The only
configu ration entry p oint for this circu it is:
$OV_CON F/ d ed up.conf
Connector dow n event correlation circuit
This correlation is m anaged by the Correlation Manager. N N M‟s first
correlator, this ECS logic w as introd u ced in V6.0. Connector Dow n
affects both netmon & APA statu s. Both netmon and APA internally
d istingu ish betw een p rim ary and second ary failu res, and p ass this
d ata to the correlation by w ay of varbind s. What this circu it d oes,
how ever, is very d ifferent at a fu nctional level w ith resp ect to netmon
versu s APA behavior.
For netmon-based events, the circu it p erform s three layers of
correlation w hich can be seen in the event brow ser as an event w ith
em bed d ed events w hich them selves have em bed d ed events (see
Figu re 10-1). At the top layer are nod e events w ith interface -level
events em bed d ed below .
For each interface event, second ary interface alarm s are su bsequ ently
em bed d ed , rep resenting cascad e failu res beyond the p rim ary interface
121
Fognet’s Field Guide to OpenView NNM
Figure 10-1
in term s of netw ork p ath. netmon bu ild s and m aintains p ath d ata on
d aem on start for every interface in the top ology. Only w ith later versions
d oes netmon issu e d ynam ic p olls based on connector statu s. For all
versions after V6.0, thou gh, netmon d ou bles (by d efau lt) sched u led p olls
to second ary interfaces.
For APA-based events, all second ary alarm s are su p p ressed , so the
circu it is instead u sed to correlate the three layers of event based on
Ad d ress events, Interface events, and N od e events (see Figu re 10-2). Like
netmon-based correlation, the nod e level is the top layer, w ith interface
events em bed d ed below and ad d ress level events em bed d ed below that.
N ote that APA-based p rim ary/ second ary d eterm ination is p erform ed by
ovet_pathengine, not ovet_poll. In ET Dynam ic view s, second ary statu s in
the top ology is set to u nknow n and in IPMAP, the statu s is u nchanged .
For netmon-based correlation, an Im p ortant N od e filter d efines nod es
that w ill alw ays be consid ered p rim ary. This is d efined u sing N N M‟s
stand ard filter d efinition langu age. The genannosrvr p rocess feed s
im p ortant nod e filter d ata to ECS. For APA -based correlation, im p ortant
nod es are configu red in the p aConfig.xm l file. Im p ortant nod es alw ays
get a statu s change w hen a d evice in the p ath to them becom es
u nreachable.
122
Event Correlation
Figure 10-2
N odeIf event correlation
This correlation is m anaged by the Correlation Com p oser, and affects
netmon statu s only. It w as introd u ced in V6.31. Also called
“Rou ter/ Sw itch H ealth,” this correlator su p p lem ents the Connector
Dow n circu it.
N od eIf su p p resses sim p le nod e d ow ns and is central to the m ajor shift
introd u ced in V6.31 from nod e-centric statu s events to Interface-centric
statu s events for netmon. N od eIf also correlates the d ynam ic netmon
p olls for neighbor interfaces also introd u ced in V6.31 (see p age 55).
N od eIf‟s effect is that OV_IF_Unknow n and OV_IF_Dow n statu s
alarm s from interfaces w ithin rou ters or sw itches are su p p ressed and
em bed d ed
if
a
corresp ond ing
nod e-level
statu s
alarm
(OV_N od e_Unknow n; OV_N od e_Dow n) is received w ithin the
sp ecified tim e w ind ow .
N od eIf su p p resses interface statu s alarm s from d evices other than
rou ters or sw itches so that these alarm s never ap p ear. The assumption is
that end-node status alarms are not desirable.
N od eIf su p p resses interface statu s alarm s from u nu sed (u nconnected )
p orts w ithin rou ters or sw itches.
123
Fognet’s Field Guide to OpenView NNM
N od eIf ad d resses the follow ing failu re scenarios that ConnectorDow n
d oesn't:
 If few er than all the interfaces on a router or sw itch fail, d ispla ys the
interface status alarm s
 If all the interfaces on a router or sw itch fail, d isplay the
OV_N od e_Dow n alarm w ith the interface alarm s nested und erneath
 If a router or sw itch failure is a second ary failure, then all second ary
events are em bedd ed , unless they pass the Im portant Nod e filter
Repeated event correlation circuit
This correlation is m anaged by the Correlation M anager. The
fu nctionality of this correlator, w hich shou ld be consid ered a legacy
circu it, is su p ersed ed by introd u ction of DeDu p correlation in N N M
6.4. The rep eated event circu it em bed s su bsequ ent m atches u nd er
original event in alarm brow ser, increm enting the correlated m ess age
cou nt. Contrast this to DeDu p , w hich d eletes and em bed s original
alarm , effectively “m oving” the original alarm to the head of alarm list.
This correlator is u sefu l for su p p ressing inform ational m essages, and is
em p loyed on the netmon-based N od e_u p event. The N od e_u p event
occu rs w hen “all interfaces are norm al or u nknow n,” so in the case
w here one interface goes d ow n and com es u p again, a N od e_u p event
m ay be rep eated ly generated .
This circu it is u sefu l for ind icating the cond ition w ithou t m oving the
m essage in the alarm brow ser each tim e the cond ition rep eats, w hich is
w hat the DeDu p correlator w ou ld d o.
PairWise event correlation circuit
This correlation is m an aged by the Correlation M anager and it ap p lied
to m ost statu s events generated by either netmon or the APA. The
behavior of PairWise is very d ep end ent on the version of N N M and
su bstantially changes betw een V6.31, V6.41, and again in V7.01. The
behavior of PairWise is also loosely cou p led to DeDu p correlation
behavior.
N N M V6.0, V6.1, V6.2:
Status alarm s are acknowledged if the parent event received w ithin 10
m inutes (Paired Tim eWind ow ). Child events are released im m ed iately to
brow ser and associated actions are im m ed iately launched . There is no
124
Event Correlation
red uction in the num ber of alarm s, as the p arent events not em bed d ed
(suppressed ).
N N M V6.31, V6.41:
Status alarm s are deleted if the parent event is received w ithin 10 m inutes
(Paired Tim eWind ow ). Child events are held (not released im m ed iately),
and should not be seen in the alarm brow ser at all. Actions are launched
only if the Paired Tim eWind ow expires or the parent event is d etected . N o
alarm s are seen at all if parent event is received w ithin the w ind ow and if
the Paired Tim eWind ow expires, the child event is released w ith the
original tim e stam p to the alarm brow ser. Subsequently, the p arent event is
em bed d ed , and should be subject to bein g d eleted by the DeDup circuit
(see below ).
N N M V7.01+:
Child events are released im m ed iately to the alarm brow ser and
associated actions are im m ed iately launched . Status alarm s subsequently
d eleted if parent received w ithin 10m . Parent event received after w ind ow
is sent to alarm brow ser, and are not em bed d ed .
D eD up event correlation
This correlator is m anaged by the Correlation Com p oser and affects
netmon & APA statu s. It w as introd u ced in V6.41+. It is an
im p rovem ent on the Rep eated Event in that it u ses p ost-p rocessing
logic that takes feed from event d atabase as op p osed to the event
stream via pmd. DeDu p d eletes and em bed s the existing m atching
alarm , “rep lacing” it w ith the latest alarm .
DeDu p w as cou p led to PairWise circu it in N N M 7.0+. Any child event
d efined in DeDu p and PairWise shou ld be d eleted by the receip t of the
p arent alarm , even after PairWise Paired Tim eWind ow exp ires. The
affected “d ow n” statu s alarm s are alw ays d eleted from the alarm
brow ser w hen m atched w ith their corresp ond ing “u p ” alarm .
This
correlator
is
configu red
via
an
external
file,
$OV_CON F/ d ed u p .conf. Uncom m ent the follow ing line to entirely
d isable this correlation:
DEDUPLICATION=OFF
Intermittent Status event correlation
This correlation is m anaged by the Correlation Com p oser, w as
introd u ced in V6.31, and affects netmon statu s only. This correlation
is
also
called
“Rou ter/ Sw itch
Interm ittent
Statu s”
or
125
Fognet’s Field Guide to OpenView NNM
“OV_Connector_Interm ittentStatu s.” Interm ittent Statu s show s
flap p ing interfaces w hose statu s w ou ld otherw ise be su p p ressed by
the PairWise correlation‟s behavior. This logic only ap p lies to
connector interfaces, and it generates a new event, OV_IF_Interm ittent
(Op enView enterp rise 58982423) to ind icate the flap p ing cond ition.
The new event is generated w hen the inp u t event occu rs
RATE_COUN T tim es w ithin RATE_PERIOD, w here by d efau lt:
RATE_COUN T = 4 in V6.31
RATE_COUN T = 5 in V6.4
RATE_COUN T = 4 in V7.0+
RATE_PERIOD = 30 m inutes
OV poller plus event correlation
This correlation is m anaged by the Correlation Com p oser, w as
introd u ced in V6.41, and affects APA statu s only. This correlator is
u nload ed by d efau lt. This correlator has the sam e fu nctionality as
Interm ittentStatu s, bu t ap p lies to the LinkDow n event and to APA
statu s events affected by PairWise correlation.
The correlator com p rises five ind ivid u al logic sets for APA connection,
interface, ad d ress, nod e events and LinkDow n trap s. The new event
generated w hen a flap p ing cond ition is d etected is of the form :
OV_APA_[IN TERFACE| N ODE| ADDR| CONN ]_Interm ittent
This alarm is generated w hen like alarm s p rocessed by PairWise
correlation rep eat RATE_COUN T tim es w ithin RATE_PERIOD,
w here:
RATE_COUN T: Default is 2 in V7.01
RATE_PERIOD Default is 30 m inutes
To enable this correlator (d isabled by d efau lt), a d d the follow ing entry
to the $OV_CON F/ ecs/ CIB/ N am eSp ace.conf file, then red ep loy the
correlators w ithin the Correlation Com p oser w ind ow :
OV_Poller_Plus=PollerPlus.fs
Multiple reboots event correlation
This correlation is m anaged by the Correlation Com p oser and is not
tied to Op enView statu s events. This correlator creates a new alarm
w hen 4 (by d efau lt) or m ore cold Start or w arm Start trap s are received
w ithin 5 (by d efau lt) m inu tes from a sp ecific SN MP -agent/ IP-ad d ress
p air. This correlator is yet another rep eat of the fu nctionally of
Interm ittent Statu s (see above). Introd u ced in V6.31, this correlator
126
Event Correlation
w as d esigned to d etect m u ltip le reboots w hich w ou ld otherw ise never
been seen p er the PairWise 6.31 behavior (see above).
Cisco chassis event correlation
This correlation is m anaged by the Correlation Com p oser and it
m onitors the follow ing CISCO-STACK-MIB trap s and generates one of
three new Op enView enterp rise alarm s if the cond ition p ersists for 10
m inu tes:
.1.3.6.1.4.1.9.5.5
.1.3.6.1.4.1.9.5.6
The Op enView trap s generated are:
OV_Chassis_Tem perature (specific 8982424)
OV_Chassis_FanFailure (specific 58982425)
OV_Chassis_Pow erSupply (specific 58982426)
The circu it requ ires that the Stack MIB is load ed on the m anagem ent
server and that the trap s are enabled for each Cisco d evice (they are
d isabled by d efau lt). See p age 187 for instru ctions on how to enable
these trap s.
Summary of event correlations affecting status
Figu re 10-3 show s som e of the interactions betw een N N M and the
event correlation engine. Events flow into the ovevent stack w ithin pmd.
There, events are d irected on p arallel tracks to d aem on that register
w ith pmd. The d aem ons register for raw , correlated , or all ou tp u t from
the ovevent stack. ECS is one of the d aem ons that register for this feed ,
bu t it is sp ecial in that it is able to feed its ou tp u t back into the ovevent
stack as inp u t.
127
Fognet’s Field Guide to OpenView NNM
Figure 10-3
128
Event Correlation
Useful ECS manager commands
The follow ing com m and s d isp lay Engine Statu s and d etails:
ecsmgr
ecsmgr
ecsmgr
ecsmgr
–info
–stats
–stats verbose
–snapshot <output_file>
The ecsmgr –stats com m and show s w hich ECS circu its are d oing
w hat w ith each event they p rocess. If there is a cou nt associated w ith
the u nd ecid ed stat, this gives the nu m ber of events that are cu rrently
being held by the circu it in som e sort of tim e w ind ow .
The ecsmgr –snapshot com m and d u m p s d etailed inform ation abou t
the states of op erating circu its. It w ill also show d etails abou t the
events that are cu rrently being held by each ECS circu it.
To externally d isable/ enable ECS circu it:
ecsmgr –disable PairWise
ecsmgr –enable PairWise
To externally change ECS Circu it, m od ify the .d s file, then:
ecsmgr –disable PairWise
ecsmgr –circuit_reload PairWise
ecsmgr –enable PairWise
Configuring ECS circuits
When N N M-su p p lied ECS circu its are configu red u sing the
Correlation Manager to change p aram eters su ch as tim e w ind ow s, etc.,
the follow ing files shou ld get u p d ated au tom atically. Som etim es,
errors are p rod u ced after changing p aram eters. If so, check these files:
$OV_CON F/ ecs/ circuits/ <circuit_nam e>.d s
$OV_CON F/ ecs/ circuits/ C/ <circuit_nam e>.param
D isable or enable ECS
ECS ru ntim e and d efau lt ECS correlators are enabled by d efau lt w hen
N N M is installed . To p erm anently d isable ECS, a d d the follow ing
sw itch to p m d .lrf (p roced u re on p age 5):
-SOV_EVENT;p0
Disable ECS on the ru nning p m d : pmdmgr -SOV_EVENT\;p0
To re-enable ECS after rem oving the -SOV_EVENT;p0 LRF flag, ru n:
pmdmgr -SOV_EVENT\;p1
129
Fognet’s Field Guide to OpenView NNM
To d isable all ECS circu its on the N N M instance u ntil the system
reboots or the ovstart is issu ed , ru n: ecsmgr –reset
To re-enable all circu its after a reset: For each .d s file in:
$OV_CON F/ ECS/ circuits: ecsmgr –enable <circuit_name>
Or, m anu ally toggle the correlations back on in the GUI.
Logging incoming ECS events
To cap tu re all events entering the ECS engine:
Turn on logging
Turn off logging:
Change log size:
ecsmgr –log_events input on
ecsmgr –log_events input off
ecsmgr –max_log_size event <Kbytes>
The log file nam e is ecsin.evt0. It is a rolling log, and is 512k by d efau lt.
It is located in:
$OV_LOG/ ecs/ 1/ ecsin.evt0 and
$OV_LOG/ ecs/ 1/ ecsin.evt1
Logging output and correlated events
To cap tu re events (inclu d ing ECS-created events) ou tp u t or d iscard ed
by the cu rrently enabled correlators:
ecsmgr –log_events stream on
ecsmgr –log_events stream off
ecsmgr –max_log_size event <Kbytes>
The log file is nam ed d efau lt_xxx.evt0. It is a rolling log, it is 512K by
d efau lt, and is located in:
$OV_LOG/ ecs/ 1/ d efault_sout.evt0
$OV_LOG/ ecs/ 1/ d efault_sout.evt1
Events w hich are d iscard ed by the stream (or su p p ressed by a circu it)
are w ritten to:
$OV_LOG/ ecs/ 1/ d efault_sd is.evt0
$OV_LOG/ ecs/ 1/ d efault_sd is.evt1
Simulate events for testing ECS logic
ecsevgen and ecsevout in $OV_CON TRIB/ ECS can be u sed to rep lay
event logs for testing the effects of ECS circu it changes. For m ore info,
see Trou bleshootingEventCorrelation.txt in that d irectory.
130
Event Correlation
ECS tracing
ECS tracing can be extrem ely verbose, bu t certain d evelop m ent
activities can‟t be su fficiently trou bleshot w ith ou t it. To enable fu ll ECS
tracing:
ecsmgr -i 1 -trace 65536
pmdmgr -D0xffffffff -SECSS\;T0xffffffff
The traces are then w ritten to $OV_LOG / p m d .trc0.
Stack tracing via pmdmgr
To tu rn on the tracing of the OV_EVEN T stack:
pmdmgr -SOV_EVENT\;T0xffffffff
This shou ld p rod u ce entries in $OV_LOG/ p m d .trc0. To tu rn on pmd
d ebu gging to trace all stacks (SN MP, etc):
pmdmgr -D0xffffffff
or
pmdmgr -D0xffffffff -SECSS\;T0xffffffff
To m ake su re pmd p roblem s are not related to ECS, before starting
pmd, renam e or m ove the ECSS stack configu ration file:
/ etc/ opt/ OV/ share/ conf/ stacks/ pm d/ ECSS.cfg
To tu rn tracing off, sim p ly restart OV, or ru n:
pmdmgr
-SOV_EVENT\;T0x0
Monitoring pmd bottlenecks (N N M 6.1 only)
This version only show ed pmd p erform ance in ovstatu s ou tp u t. This
rep orting w as later rem oved , as it w as a d evelop m ent d ebu gging
sw itch that w as accid entally left on. Ru nning „ovstatus pmd‘ w ou ld
rep ort:
pmd RUNNING Statistics: cce=3, rqts=0, memory unfreed=6
"rqts" is nu m ber of p end ing op erations in the pmd event p rocessing
qu eu e. A valu e above 50 cou ld m ean bottlenecks.
Correlation Composer tracing
Follow these step s to enable Com p oser tracing.
1. Enable tracing w ithin the Com poser GUI
131
Fognet’s Field Guide to OpenView NNM
2.
3.
4.
5.
cd $OV_CONTRIB/ ecs/ CO
Run: ecsmgr –fact_update Composer CompTraceOn.fs
Run: pmdmgr –SECSS\;T0xffffffff
Tracing output is w ritten to $OV_LOG/ pm d .trc0
To d isable Com p oser tracing:
1. Run: pmdmgr –SECSS\;T0x0
2. Run: ecsmgr –fact_update Composer CompTraceOff.fs
3. Disable tracing w ithin the Com poser GUI
132
11. SNMP Functions
Up on installation, N N M im p lem ents a p rop rietary SN MP agent
infrastru ctu re (Em anate) d esigned to accom m od ate m u ltip le SN MP
agents on a single server (the N N M m anagem ent station). This allow s
N N M‟s sp ecial agent to co-exist w ith OS-oriented agents, therefore
allow ing for extensibility. The Em anate m aster and su bagents are OEM‟d
from SN MP Research, Inc. as p art of the N N M p ackage. SN MP Research
is also the p rovid er of the SN MP Secu rity Pack that is requ ired for N N M
to su p p ort SN MPv3 agents. SN MPv3 p rovid es strong secu rity su p p ort,
am ong other ad vanced SN MP featu res.
As of N N M 7.5, the SN MP Secu rity Pack com es bu nd led w ith the AE
version, bu t only the Em anate agent is installed by d efau lt. The Secu rity
Pack inclu d es an ap p lication to rem otely configu re SN MPv3 u ser
cred entials, access rights, and keys on the m anaged nod es. The SN MP
Secu rity Pack requ ires licensing throu gh SN MP Research and it is
available for all p latform s and versions su p p orted by N N M. N N M
7.5/ Secu rity Pack integration also p rovid es su p p ort for ET d iscovery via
SN MPv3. Installation of the SN MP Secu rity Pack rep laces the Em anate
agent w ith SN MP Research‟s m ore ad vanced Brass agent.
SN MP versions and history
SN MPv1 and V3 are fu ll IETF stand ard s. SN MPv2 never achieved
stand ard statu s. There are three versions of non-stand ard SN MPv2
that have been im p lem ented by a variety of vend ors: V2C, V2U, and
V2*. SN MPv2C is su p p orted natively in N N M. SN MPv1 ap p eared in
1988 u nd er RFC‟s 1065, 1066, and 1067. It is still w id ely u sed bu t has
been criticized for its u se of client au thorization via cleartext
com m u nity strings. SN MPv2, som etim es also referred to as SN MPv2P,
w as initially introd u ced throu gh RFC‟s 1441 and 1452. It offered
im p rovem ents of SN MPv1 and im p lem ented a p arty-based secu rity
system w hich som e view ed as overly com p lex. SN MPv2C (as
p rop osed in RFC‟s 1901 thou gh 1908) introd u ced a com m u nity-based
secu rity m od el w hich com p rised SN MPv2 w ithou t the controversial
133
Fognet’s Field Guide to OpenView NNM
new SN MPv2 secu rity m od el, u sing instead the fam iliar com m u nitybased secu rity schem e of SN MPv1.
SN MPv2U (as p rop osed in RFC 1909) attem p ted to offer greater
secu rity than SN MPv1, bu t w ithou t incu rring the high com p lexity of
SN MPv2. A variant of this w as com m ercialized as SN MPv2* and the
m echanism w as eventu ally ad op ted as one of tw o secu rity fram ew orks
in SN MPv3. SN MPv3 as d efined in RFC‟s 3411 and 3418 achieved fu ll
stand ard statu s in 2004. It encom p asses strong secu rity, p rovid es
rem ote configu ration, has ad vanced scalability and d istribu tion
featu res, and is com p atible w ith d ep loyed p reviou s SN MP versions.
Configuring SN MP community strings on N N M agent
Com m u nity strings for N N M‟s cu stom SN MP agent (Em anate) are
configu red in the snm p d .conf file.
On UN IX, this file is in :
On Wind ow s , this file is in:
/ etc/ Snm pAgent.d
%OV_CON F%\ SN MPAgent
N ote that Em anate d oes not allow the w rite and read com m u nities to
be set to the sam e string.
Configuring SN MP versions
N N M su p p orts SN MPv1 and SN MPv2C, and SN MPv3 w ith the SN MP
Secu rity Pack. To sp ecify SN MPv2C as the p rotocol to u se for a
p articu lar agent, u se xnmsnmpconf w ith the –setV op tion. To
d eterm ine the version, u se:
xnmsnmpconf –getV <target>
Mu ltip le p rotocols can be sp ecified , for exam p le:
xnmsnmpconf –setV <target> 1,2C
Configuring multiple SN MP agents on a single node
Som e d evices (e.g. servers) can su p p ort m u ltip le SN MP agents if those
agents are bou nd to non-conflicting p orts. N N M su p p orts only one
SN MP p ort p er IP H ostnam e, thou gh. If the target system had m u ltip le
ad ap ters, or if virtu al IP ad d resses w ere assigned to an interface,
sep arate SN MP agents cou ld w ork if each ad d ress w ere assigned a
sep arate DN S nam e so that N N M cou ld d iscover them as thou gh they
w ere tw o nod es. This is actu ally how SN MP agent sim u lators like
Gam bit‟s MIMIC w ork. If the system is the N N M server, the Em anate
extension agent can be u sed to ad d SN MP agents. On Wind ow s,
134
SNMP Functions
wpaagt load s extension agents. The ad ap ter read s this registry key to
find the extension agent DLLs to load :
H KLM\ SYSTEM\ CurrentControlSet\ Services\ Paramters\ ExtensionAgents
On UN IX, the extension agent is naaagt and is configu red via the file:
/ etc/ Snm pAgent.d/ naa.cnf.
More info in can be fou nd in the agent‟s resp ective m an/ ref p ages. A
creative w ay to su p p ort m u ltip le m onolithic agents: is to configu re
som e extensible agent that is bou nd to p ort 161 to p roxy requ ests to the
agents that are bou nd to other p orts. For m ore on extensible agents, see
p age 146.
SN MP master agent sw itches
To enable tracing on UN IX p latform s, first stop the d aem on u sing:
/ sbin/SnmpAgtStart.d/K03SnmpMaster
To enable tracing on Wind ow s p latform s, first stop the N T m aster
agent Service. For both p latform s, once the agent has been stop p ed ,
ru n:
snmpdm -apall -start
The trace file shou ld be either / var/ ad m / snm p d .log or
/ tm p / snm p d .log. For SN MP Packet d u m p ing, see the –hexdump –
vbdump op tions. To re-install, ru n:
snmpdm –stop –remove
snmpdm –install
SN MPv3 configuration using SN MP security pack
Once the SN MP Secu rity Pack is p rop erly installed , configu re the
SN MPv3 agents throu gh the stand ard SN MP Configu ration GUI
(xnmsnmpconf) and enter the au thentication d ata u sing the keyw ord as
p art of the com m u nity string, e.g:
"3A;AuthPassword/AuthUser"
SN MPv3 offers three levels of secu rity w here Au thorization (Au th)
equ ates to u ser au thentication and Privacy (Priv) equ ates to d ata
encryp tion. The Secu rity Pack secu rity m od e p refixes (like that u sed in
the exam p le above) corresp ond to the levels as follow s:
3N;
noAuth/ noPriv:
supply usernam e only, e.g. “guest”
135
Fognet’s Field Guide to OpenView NNM
3A;
3P;
Auth/ noPriv:
Auth/ Priv:
supply pass phrase (user/ pass)
supply pass phrase (user/ pass)
SN MPv3 Security Pack installation
Follow these step s to install the SN MPv3 Secu rity Pack:
1. Install the Pack (UNIX target is / opt/ Snm pri/ SecurityPack)
2. Make sure the process brassd is running: ovstatus –c brassd
3. Ed it the configuration files for the brassd daem on:
/ etc/ srconf/ agt
for agent
/ etc/ srconf/ m gr for the manager sid e of SN MPv3 process
4. Configure the usernam e/ password for SN MPv3
5. Configure the d evices to allow SNMPv3 protocol
N ote that the Pack rep laces the Em anate Agent w ith brassd.
Log file:
Comm and s d ir:
MIB d irectory:
/ opt/ Snm pri/ SecurityPack/ log
/ opt/ Snm pri/ SecurityPack/ brass/ bin
/ opt/ Snm pri/ SecurityPack/ m ib
Exam p les of SN MPv3 qu eries:
snmpwalk -v 3 -l authNoPriv -a MD5 -u usr -A pwd <node>
snmpwalk –c 3A;authPassword/authUser <target>
Configuring and controlling trap destinations
The N N M server attem p ts to set itself as the trap d estination for any
d evice it d iscovers. Most SN MP agents are fairly qu iet w ith resp ect to
the am ou nt of SN MP trap s they generate, bu t there are d efinitely
excep tions. H P netw ork p rinters, for exam p le, can be very chatty by
d efau lt. To globally p revent N N M from setting itself as an agent‟s trap
d estination, u se the “-N ” LRF flag to netmon (see LRF Proced u re on
p age 5). To p revent N N M from setting itself as an agent‟s trap
d estination for a class of d evices, d isable SN MP d iscovery u sing the “ I” flag in the oid_to_type file (see p age 7).
Prior to V6.0, netmon u sed (accord ing the netmon m an/ ref p age) a
m echanism that “byp assed norm al SN MP au thentication m easu res.”
After V6.0, netmon u ses the set com m u nity string p rovid ed in the
SN MP configu ration (xnmsnmpconf) to attem p t to set the m anagem ent
server‟s IP ad d ress as the trap d estination for any d iscovered SN MP
agent.
136
SNMP Functions
SN MPv3 and VLAN information
There is a know n p roblem w ith N N M 7.53 and earlier versions of
N N M w here if only SN MPv3 qu eries are m ad e to the d evice (via ET's
hook into the BRASS agent), in m ost cases, the VLAN tables of su ch a
d evice can not be qu eried d u e to the fact that the sp ecial character (@)
that ET internally ap p end s to the configu red com m u nity string d oes
not w ork for the sp ecial com m u nity string configu ration entries
need ed for u se by the BRASS agent.
Controlling the N N M server SN MP address
Control over w hich interface on the N N M server is set for trap
d estinations and ou tgoing SN MP requ ests can be changed u sing the
$OV_CON F/ ov.conf file. See the ov.conf m an/ ref p age for d etails.
Controlling an agent’s SN MP address
The SN MP ad d ress u sed for an agent is d eterm ined d u ring netmon
d iscovery u sing the algorithm listed on p age 26. By d efau lt, netmon
tests other ad d resses if the d iscovered ad d ress fails to resp ond . This
behavior is controlled by the –k pickSnmpAddrPolls op tion as
d iscu ssed on p age 43.
For APA p olling, the SN MP Ad d ress is initially d eterm ined by netmon
d iscovery and is affected by –k pickSnmpAddrPolls as m entioned
above, then fu ther refined throu gh p aConfig.xm l confiru ration
settings. To force a nod e to u se an SN MP Ad d ress other than the one
p icked by netmon d iscovery, see the p roced u re on p age 44.To control
the p referred m anagem ent ad d ress op tions in the APA, see p age 92.
Community string discovery
This is a u sefu l featu re, bu t it is also a p ossible sou rce of p erform ance
d egrad ation if it is u sed too generally in highly scaled installations.
Configu re com m u nity string d iscovery u sing:
$OV_CON F/ netm on.cm str
Prior to V 7.53 Interm ed iate Patch 19, the hash character (#) w as an
invalid character in com m u nity strings. After that p atchm that
character is allow ed , bu t it m u st be contained w ithin d ou ble qu otes.
Try to lim it the u se of w ild card s, p articu larly in highly -scaled
environm ents. See p age 52 for inform ation abou t setting SN MP
137
Fognet’s Field Guide to OpenView NNM
tim eou ts and retries. In the follow ing exam p le, p u blic1 is u sed for
d evices in the 10.2 netw ork, and p u blic2 is tried if p u blic1 tim es ou t:
"public1", "public2" : 10.2.*.* : : :
CERT advisory CA-2002-03, SN MP vulnerabilities
In 2002, OUSPG ran tests and fou nd issu es w ith SN MP requ est
hand ling and w ith SN MP trap hand ling in both agents and m anagers.
In essence, the ad visory said w hat everybod y everyw here all read y
knew : that SN MPv1 is insecu re and its u se can exp ose system to
exp loitation. It fu rther fou nd sp ecific vu lnerabilities for a lim ited set of
SN MP agents that cou ld lead to DOS attacks, bu ffer u nd erru n exp loits,
and other nastiness. The sou rce of these w as fou nd to be in the vend orsp ecific fu nctions w ritten to p arse ASN .1 form atted MIB d efinitions,
bu t fortu nately not in ASN .1 itself. These vu lnerabilities w ere
im m ed iately ad d ressed by m ost vend ors and by H P throu gh p atches
to N N M 6.2 and other H P p rod u cts. More info on this at:
w w w.cert.org/ ad visories/ CA-2002-03.htm l
w w w.cert.org/ tech_tips/ snm p_faq.htm l
SN MP PD U size limitations
This is a concern w hen u sing d ata collections. When there are m any
collections configu red , there m ay be excessive fragm entations
attribu table to N N M SN MP op erations. RFC1157 states that "An
im p lem entation of this [SN MP] p rotocol need not accep t m essages
w hose length exceed s 484 octets. H ow ever, it is recom m end ed that
im p lem entations su p p ort larger d atagram s w henever feasible." N N M
has no su ch lim itations, bu t som e agents m ay ad here to this low er
lim it. In p ractice, this is an excep tion and not the ru le.
Accou nting for IP and UDP encap su lation, the theoretical m axim u m
SN MP PDU size is 65467 bytes (m ax UDP d atagram size - m ax IP
head er length - UDP head er length). It is very likely, how ever, that
su ch an IP p acket is fragm ented , increasing the chance of losing the
entire PDU, since m any MTUs for IP are set at 1500 bytes. In this case,
the m ax PDU w ou ld be cu t into over 40 fragm ents, the loss of one of
w hich d u e to a collision, for exam p le, m eans the loss of the entire PDU.
For this reason, it is recom m end ed that SN MP PDUs be lim ited to ju st
u nd er the m inim u m MTUs in the rou te betw een the server and target.
In N N M, PDU size can be d ictated throu gh the “ -m ” LRF (see p age 5)
op tion to snmpCollect and m onitored via the ovstatus –v
snmpCollect com m and . Also, there w as a know n secu rity p roblem
138
SNMP Functions
w ith invalid ly-form ed trap s w hose PDU sizes exceed ed 4.5k bytes
w ith N N M version 6.2 and p rior. That sp ecific issu e w as ad d ressed
via p atches. More on this CERT ad visory-related issu e can be fou nd
below . N ote that SN MPv3 ad d ressed som e of the above-m entioned
lim itations d irectly.
ov t rapd daemon not starting (Window s)
There is know n p roblem w ith N N M 6.x version on Wind ow s w h ich is
cau sed by the fact that Wind ow s services start in rand om ord er, and
ovtrapd is d ep end ent on WinSN MP service. The error that
interm ittently occu rs on system boot u p is:
ovtrapd is not running. Error message is:
winsnmp snmpregister() failed: 100. exit 1
To resolve the p roblem , configu re ovtrapd to listen for trap s d irectly on
p ort 162/ u d p and 0xc900f/ ip x and not register w ith the Wind ow s
SN MP trap service. Do this by configu ring ovtrapd to u se the "-W" LRF
sw itch (see p age 5).
N N M w ith SMS or other SN MP tools (Window s)
SMS (and several other tools w ith SN MP m anagem ent cap abilities)
rep laces w snm p 32.d ll w ith their ow n versions or installs the Microsoft
version of w snm p 32.d ll. When N N M is installed , it installs a cu stom
version of w snm p 32.d ll. If there is a conflict, let the new er ap p lication
u se its p referred version of w snm p 32.d ll and u se the –W LRF sw itch to
ovtrap d .lrf. See p age 5 for LRF p roced u re.
Binding SN MP trap reception to ports other than 162
ovtrapd can be m ad e to listen to other p orts for SN MP trap s. In the
services file (/ etc/ services on UN IX,
or on Wind ow s:
..\ w ind ow s32\ d rivers\ etc\ services), there is an SN MP trap entry set
to p ort 162 (UDP), and this can be changed to a d ifferent UDP p ort.
ovtrapd d aem on follow s the p ort bind ing of SN MP. pmd, how ever, u ses
p ort 162 (TCP) for talking to other cop ies of N N M, so changing the
SN MP trap receiving p ort to any other valu e can break the rem ote pmd
com m u nications in environm ents u sing DIM or N N M to N N M event
forw ard ing.
SN MP manager command line utilities
All the com m and s below are tied to N N M libraries, so they w ill only
w ork on the N N M server:
139
Fognet’s Field Guide to OpenView NNM
snmpget
snmpwalk
snmpset
snmpbulk
snmpnotify
snmptrap
xnmsnmpconf
snmpcollect
xnmbrowser
xnmtrap
xnmloadmib
xnmevents
xnmgraph
mibtable
rnetstat
Query a single MIB variable
Query an entire MIB table
Set or change a w riteable MIB variable
Query a MIB table via SN MPv2 GetBulk
Issue an SN MP trap or inform request
(d eprecated by snm pnotify)
SN MP configuration utility
Data collector background daem on
MIB brow ser
Event configuration utility
MIB load er utility
N ative event brow ser
MIB graphing utility
SN MP table data d isplay utility
SN MP netw ork statistics utility
N ote that m any SN MP m anager p ackages u se the sam e nam es for
these p rogram s. For exam p le, Red H at Linu x AS 2.1 inclu d es the u cd snm p p ackage w hich inclu d es several u tilities nam ed the sam e. If
ru nning N N M u nd er AS 2.1, the u cd -snm p u tilities w ith the sam e
nam es as the N N M u tilities w ill ru n before the N N M su p p lied
com m and s becau se / u sr/ bin is ap p ears in the p ath d efinition before
/ op t/ OV/ bin. Sim p le u p d ate the $PATH variable so / op t/ OV/ bin is
before / u sr/ bin to get arou nd this issu e. It m ay be necessary to alo
u p d ate the $MAN PATH variable in a sim ilar fashion.
Strip OID and type data from SN MP query output
This w orks on both UN IX and Wind ow s p latform s. N orm ally, SN MP
qu eries retu rn a lot of garbage, e.g.:
C:\>snmpget patchy ifEntry.ifDescr.2
interface interfaces.ifTable.ifEntry.ifDescr.2 : DISPLAY
STRING- (ascii): 3Com 3C920 Integrated Fast Ethernet
Controller (3C905C-TX Compatible)
The follow ing com m and strip s the OID and d atatyp e d ata:
C:\>snmpget patchy ifEntry.ifDescr.2 |
%OV_BIN%\bin\perl\bin\perl -p -e "s /.*: //g;"
3Com 3C920 Integrated Fast Ethernet Controller (3C905CTX Compatible)
Perl script: output a node’s IP address table via SN MP
This is u sefu l for feed ing other tools or for trou bleshooting DN S issu es.
It is also a good tem p late for u sing Perl and N N M‟s SN MP qu ery tools
for bu ild ing rep orts, etc.:
#! /usr/bin/perl
$host = $ARGV[0];
$mib2 = '.iso.org.dod.internet.mgmt.mib-2';
140
SNMP Functions
$ipt = 'ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex';
@cmd = `/opt/OV/bin/snmpwalk $host $mib2.$ipt`;
foreach $line (@cmd) {
$line =~ /^$ipt\.(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/;
$ip_addr = $1;
print "$ip_addr\n";
}
exit;
Scrip t ou tp u t exam p le:
peasoup # ./snmpIPtable.pl sunny
127.0.0.1
192.168.1.107
192.168.1.108
peasoup #
SN MP w eb resources
General SN MP Info:
w w w.snm plink.org
Many MIBS:
w w w.oid view.com / m ibs/ d etail.htm l
Many MIBS:
w w w.m ibd epot.com
Many MIBS:
assure24.com/ databases/ snm p -mib/ private
SN MP OID Registry:
w w w.iana.org
All OIDS:
w w w.iana.org/ assignm ents/ enterprise-num bers
OID translation:
jaguar.ir.m iam i.ed u/ ~marcus/ snm ptrans.htm l
Cisco specific info:
w w w.cico.com/ warp/ public/ 477/ SN MP/ m ibcom pilers.htm l
N N M d evice and protocol support:
w ww .openview .hp.com/ prod ucts/ nnm et/ support/ d evice_support.htm l
N N M d evice support:
openview.hp.com/ prod ucts/ nnmet/ support/ d evice_requirem ents.htm l
141
12. Systems Management
Several facilities exist w ithin N N M to help m anage system resou rces
su ch as d isk sp ace and system p erform ance m etrics, bu t N N M is not a
su bstitu te for a fu ll fu nction d istribu ted system s m anagem ent tool.
Syslog integration facility
Any system or d evice that u ses the syslog facility (RFC 3164, 3195) can
p rovid e a feed to this light-w eight version of H P‟s Op enView
Op erations (OVO) logfile encap su lator agent. This featu re w as
integrated into AE versions of N N M starting w ith version 7.0 and it is
only su p p orted on N N M ru nning on UN IX p latform s. If a syslog
consolid ator is requ ired for Wind ow s environm ents, several third
p arty p rod u cts are available. Kiw i is a p op u lar choice.
The syslog integration featu re is covered in d etail in Section 6 (Section
7 for V7.53) of the Gu id e to Using Extend ed Top ology u ser m anu al.
The V7.53 d ocu m entation is m ore extensive than p reviou s coverage.
Also, help is available at the follow ing URL:
http:/ / <nnm server>:3443/ OvCgi/ OvWebH elp.exe?Content=slgref
If APA is enabled , certain event correlation logics w ill u se syslog
events to increase the intelligence of the APA. Investigate w hat syslog
events affect the APA by lau nching the correlation com p oser u sing the
com m and ovcomposer –m o. Exam ine the OV_Poller* correlators.
These correlations can be extend ed by ad d ing a d d itional syslog events
to them in ord er to trigger APA p olls, etc.
The agent converts syslog m essages to N N M events and u ses the sam e
p rop rietary configu rable m essage p attern m atching exp ression
anchoring langu age u sed in OVO. To activate the N N M syslog facility
in “N N M Dep loym ent Mod e”, w hich m eans the syslog facility w ill
w ork ind ep end ently of a p eer OVO installation, the follow ing
generalized step s are requ ired :
142
Systems Management
1.
2.
3.
4.
5.
6.
If running H P-UX 11.0, make sure DCE is installed .
If running N IS, ad d opc_op user and opcgrp group on the NIS server.
Otherw ise, ad d opc_op and opcgrp locally
Use $OV_BIN/ovsyslogcfg to configure trap mappings
Run: setupSyslog.ovpl -standalone -deploy
Test
For exam p le:
Solaris:
logger -p user.err %LINEPROTO-5-UPDOWN: Line
protocol on IF test2, changed state to down
H P-UX:
logger %LINEPROTO-5-UPDOWN: Line protocol on
Interface test2, changed state to down
Use the logfile to trou bleshoot syslog configu ration or installation:
$OV_PRIV_LOG/ setupSyslog.log
To configu re a Cisco d evice to log to the N N M server:
(set) logging server <nnmserver ip>
(set) logging server enable
(set) logging level 5 all
Cisco logging levels:
0
1
2
3
4
5
6
7
emergencies
alerts
critical
errors
warnings
notification
informational
debugging
In som e cases, m essages m ay be logged to a d ifferent filenam e su ch as
local7log instead of being logged to th e syslog or m essages file. If this
is the case, m od ify syslog.conf to p oint local7 t o syslog, and the d o a
kill –HUP <PID> on the syslogd PID.
The follow ing file is w ritten to by the syslogTrap facility and this file
can grow w ithou t bou nd s:
/ var/ opt/ OV/ tm p/ OpC/ m sgagtd f
To lim it the size of that file and the other qu eu e files as w ell, enable
Op C m essage qu eu e bu ffering by ad d ing the below entries to the
op cinfo configu ration file in / op t/ OV/ bin/ Op C/ install:
OPC_BUFLIMIT_ENABLE TRUE
143
Fognet’s Field Guide to OpenView NNM
OPC_BUFLIMIT_SIZE 10000
OPC_BUFLIMIT_SEVERITY critical
Managing systems w ith HOST RESOURCES MIB
There are H OST-RESOURCES-com p liant (RFC 2790) agents that are
available both com m ercially and for free. Som e su p p ort extensions that
d o neat things like logfile scrap ing. Vend ors inclu d e H P, SN MP
Research, and Concord (CA). w w w .net-snm p .org p rovid es a com p lete
agent as w ell as SN MP m anagem ent tools for free.
For H OST-RESOURCES MIB su p p ort on the N N M server itself, the
p u rchase of SN MP Research‟s CIAgent is requ ired becau se the agent
has to be a su bagent to the Em anate M aster agent for that one
p articu lar host. An alternative is to configu re an SN MP agent that
su p p orts H OST-RESOURCES to u se a p ort other than UDP 162. Be
su re to sp ecify the d esignated p ort w hen m aking qu eries to this agent
u sing the GUI or com m and line tools.
Monitoring processes using HOST RESOURCES MIB
Once an SN MP agent is in p lace that su p p orts H OST-RESOURCES,
one can set u p a d ata collection on the N N M server. For each instance
of hrSWRu nN am e that corresp ond s to the p rocess (as rep orted in the
corresp ond ing instance of hrSWRu nStatu s), configu re the collection
u sing the follow ing p aram eters: d on't store, check threshold s, and
configu re a threshold event that rep orts if the valu e is > 1. The OID for
hrSWRu nN am e is:
.1.3.6.1.2.1.25.4.2.1.2.
Monitoring disks using HOST RESOURCES MIB
Once an SN MP agent is in p lace that su p p orts H OST-RESOURCES, on
N N M, set u p a MIB Exp ression (see p age 149) p er below and then set
u p a d ata collection (p age 147) u sing that exp ression.
FreeDiskMB \
―( hrStorageSize–hrStorageUsed) times \
hrStrorageAllocationUnits\
divided by 1024*1024
― \
1.3.6.1.2.1.25.2.3.1.5 \
1.3.6.1.2.1.25.2.3.1.6 – \
1.3.6.1.2.1.25.2.3.1.4 * 1048576 /
Managing systems using RD MI (Window s)
Wind ow s versions of N N M su p p ort RDMI m anagem ent throu gh the
ovcapsd d aem on. RDMI is H P‟s p rop rietary extension to access DMI
144
Systems Management
m anagem ent inform ation rem otely. Only som e of H P‟s hard w are lines
su p p ort RDMI. When N N M d iscovers any d evice, it tests for RDMI
cap ability and sets the isRDMISu p p orted cap ability flag. Use the
search fu nction to list RDMI su p p orted d evices that have been
d iscovered . The em bed d ed DMI brow ser can then be u sed on those
boxes. DMI as com p ared to SN MP:
- The DMI Client is roughly equivalent to an SN MP Manager
- The DMI Service Provid er is equivalent to th e SN MP Agent
- The DMI MIF is equivalent to the SN MP MIB
- A DMI Event is equivalent to an SN MP Trap
Forw arding Window s events to N N M as SN MP traps
The Wind ow s com m and line u tility evntcmd (or its GUI-based
equ ivalent evntwin) can be u sed to configu re events to be forw ard ed
to N N M via SN MP. The trap s com e to N N M u nd er the enterp rise
1.3.6.1.4.1.311.1.13.1. Best bet is to m anu ally create a new enterp rise for
these trap s rather than try to chase d ow n any MIBs that m ight exist
(they d on‟t).
Forw arding N N M events to Window s event log
Use the follow ing Perl cod e snip p et as p art of an action callback scrip t
that p asses N N M event d ata to a Wind ow s Event Log:
Use Win32::EventLog;
my $ELog;
my %event=(
'EventID',100,
'EventType',EVENTLOG_WARNING_TYPE,
'Category',NULL,
'Strings','This is a String',
'Data','THIS IS DATA',
);
$ELog = new Win32::EventLog( 'MyScript' )||die $!;
$ELog->Report(\%event) || die $!;
Managing systems using Solstice enterprise agents (SEA)
Defau lt installations of Solaris d o not have SN MP agents enabled . SEA
ship s free w ith m ost versions of SUN ‟s op erating system s and
im p lem ents a SN MP m aster/ su bagent architectu re sim ilar to Em anate
that su p p orts MIBII, DMI, and an SN MP-DMI cross m ap p er. For m ore
inform ation, or to d ow nload see:
w w w.sun.com / software/ entagents/ features.xm l
145
Fognet’s Field Guide to OpenView NNM
Accessing WMI data via SN MP
The third p arty SN MP Inform ant sharew are p rod u ct p rovid es a WMI
to SN MP agent that installs p rop rietary MIB extensions on Wind ow s
system s. It requ ires that the Microsoft SN MP agent is installed and that
the SN MP Inform ant MIB is load ed on the N N M server. The basic
agent w ith visibility to arou nd 60 m etrics is free, and extensions for
m ore d etailed WMI objects are available throu gh the fu lly licen sed
version.
N N M server system management via SN MP (UN IX)
The Em anate SN MP m aster agent and inclu d ed su bagents ship p ed
w ith N N M for H P-UX and Solaris p rovid e H P p rop rietary system s
m anagem ent extensions. Access to these can be fou nd via the ovw
p erform ance m enu as “CPU Load ” and “Disk sp ace.”
Extensible SN MP agents
These agents p rovid e a p latform for d evelop ing cu stom SN MP objects
and MIB trees. Som e of these agents p rovid e facilities and featu res that
go beyond sim p le SN MP agent d evelop m ent. H P sells su ch an agent
as d o other N MS vend ors. The N et-SN MP agent (w w w .net-snm p .org)
is very p op u lar as w ell and also p rovid es SN MP get, set, and trap
generation com m and s. SN MP Research‟s CIA Agent is also a p op u lar
and com p rehensive solu tion, as is Concord ‟s System Ed ge solu tion.
146
13. Data Collection and Thresholds
SN MP agents are d esigned to hold very sim p le d ata typ es. This allow s
the agents to be very efficient, bu t m akes the job of the SN MP m anager
very hard . Excep t in the case of RMON , SN MP agents d o not track
trend s or keep history of d ata. Interface traffic, for exam p le, is held in the
SN MP agents as cou nter valu es that sim p ly increm ent w hen traffic
p asses throu gh.
When the cou nter fills u p , it resets to zero. N N M‟s snmpCollect
backgrou nd d aem on p olls MIB valu es, d eterm ines the d ifference
betw een the cu rrent valu e and the last p olled valu e, d iscard s any cou nter
w rap s or resets, then record s the d elta in the snmpCollect d atabase on the
m anagem ent server.
Interface u tilization , p erhap s the m ost com m only u sed N N M
p erform ance m etric, is not available from d irect SN MP agents. To rep ort
this, SN MP m anagers m u st p oll ifInOctets, ifOu tOctets, and ifSp eed ,
then ad d the cou nter valu es and m u ltip ly by 8 to get the total nu m ber of
bits before d ivid ing by ifSp eed , and then m u ltip ly the resu lt by 100 to
obtain a p ercentage, as follow s:
(ifInOctets + ifOutOctets) × 8
───────────────────── × 100
ifSpeed
Math that com bines MIB variables is p erform ed in N N M by the MIB
Exp ression featu re and is configu red in the m ibExp r.conf file. N ote that
SN MP valu es for interface link sp eed m ay not be accu rate. Also, the
form u la exp ressed above is fine for half d u p lex Ethernet interfaces, bu t
others, like fram e relay and serial lines or fu ll d u p lex FastEthernet or
GigabitEthernet interfaces carry fu ll d u p lex traffic. ifTyp e can tell them
ap art, bu t snmpCollect isn‟t sm art enou gh to d istingu ish half d u p lex from
fu ll d u p lex lines.
Som e of this confu sion w as m itigated by the ad d ition of the ovexp rgu ru
in V6.01. This w rap p er for MIB d ata and MIB exp ression d ata p erform s
qu eries to d eterm ine w hat MIB variables are su p p orted before the d ata
147
Fognet’s Field Guide to OpenView NNM
collector actu ally goes off and collects them , and it has som e lim ited
abilities to d istingu ish collections by ifTyp e.
xnmgraph is the foregrou nd p rocess that d isp lays SN MP d ata from
history, real tim e feed s, or both. It is d esigned to read raw d ata from the
snmpCollect d atabase. Data from snmpCollect is exp orted to the d ata
w arehou se for trend rep orts.
D ata collection best practices
Review the d ata collections that are set by d efau lt w hen N N M is
installed . Dep end ing on the version, som e collections m ay be very
intensive for certain d evices. The N N M versions closer to V6.0 had the
m ost liberal settings for these collections.
In general, the d efau lt collections are in p lace to su p p ort the rep orting
fu nction. Disabling SN MP d ata collections d irectly in the d ata
collections and threshold s configu ration GUI can affect certain rep orts.
Conversely, d isabling rep orts u sing the rep orter GUI d oesn‟t
necessarily d isable the u nd erlying d ata collections. When setting
threshold s, a nu m ber greater than tw o is recom m end ed as the valu e of
“for X consecu tive sam p les.” Most SN MP-based d ata tend s to be
“sp iky” by natu re and this gu arantees that only su stained threshold
violations generate threshold alarm s.
Do not set the p olling intervals too tight. Generally, 15 m inu te, 30
m inu te, and 1 hou r sam p ling intervals are best. While tighter p olling
intervals tend to m ore accu rately gau ge actu al trend s by m ore
accu rately cap tu ring d ata arou nd cou nter w rap s and better isolating
d ata sp ikes, the best p ractice is to u se tighter intervals on only a sm all
set of collections w hich are intend ed to be tem p orary w here the focu s
is on sp ecific p roblem s.
Too m any collections w ith tight intervals lead to p erform ance issu es
that u ltim ately d egrad e all collections‟ integrity d u e to increased failed
d ata collection p oints from tim eou ts, etc.
If%util (or other % expression) is greater that 100%
Typ ically, this is becau se the interface being m onitored is ru nning at
fu ll d u p lex. The If%u til MIB exp ression is d esigned for half d u p lex
lines only. The next m ost com m on cau se of this is that ifSp eed is set
incorrectly at the agent. To verify the sp eed as rep orted by SN MP,
select the nod e and select the Configu ration ->N etw ork Configu ration >Interface Prop erties m enu item in ovw. Another cau se m ay be that the
148
Data Collection and Thresholds
rou ter is com p ressing d ata before send ing it over WAN links. In this
last case, the SN MP agent is record ing the incom ing d ata before
com p ression and/ or the ou tgoing d ata after d ecom p ression.
Utilization m ay exceed 100% m ight be d u e to the p resence of 64-bit
cou nters in the MIB. See section below on exhau sted 32-bit cou nters in
the p resence of 64-bit cou nters.
On fram e relay links, the ifSp eed is typ ically the line CIR, w hich is
u su ally less than the m axim u m bu rst rate for the link, so it is not
im p ossible to have u tilizations that exceed 100% on these typ es of
interfaces.
Yet another p ossibility is that inbou nd traffic and ou tbou nd traffic are
not sam p led sim u ltaneou sly. This is d u e to nu m erou s factors affecting
the snm p Collect p oller and the d ifference cou ld be several second s,
p articu larly w hen line u tilizations ap p roach 100%. The u nd erlying
MIB d ata is accu rate, bu t inaccu racy is introd u ced becau se the tim e
intervals u sed to calcu late the d eltas are fixed , yet the actu al sam p led
d ata d oesn‟t corresp ond exactly to those intervals.
Typ ically, how ever, these tim ing issu es only accou nt for m inor
d ifferences in the rep orted u tilization. Also, op ening u p the p olling
interval red u ces the chance for this sort of error. Finally, som e p eop le
are d eterm ined to som ehow com bine inbou nd and ou tbou nd
u tilization on fu ll d u p lex links. While setting u p an exp ression that
su m s the tw o rates and then d ivid es by tw o shou ld never exceed 100%,
bu t this is a p oor p ractice and can hid e real p roblem s.
For exam p le, if inbou nd is p egged at 100%, and ou tbou nd is averaging
20%, it w ou ld never be d etected since the exp re ssion w ou ld rep ort 60%
“average” u tilization. The best p ractice is to sep arate the inbou nd and
ou tbou nd collections on Fu ll Du p lex links. The best solu tion w hen
d ealing w ith m ixed d u p lex/ half d u p lex interface environm ents to u se
N N M‟s new er ovexprguru featu re, w hich is covered in m ore d etail
below .
MIB expressions using mibExpr.conf and mib.coerce
As illu strated above, sim p le SN MP qu eries have to be com bined to
m ake u sefu l grap hs. $OV_CON F/ m ibExp r.conf allow s the creation of
MIB exp ressions. Math in these form u las u ses RPN (Postfix notation).
See the m an/ ref for m ibExp r.conf
149
Fognet’s Field Guide to OpenView NNM
One requ irem ent of creating MIB exp ressions is that MIB valu es u sed
in those exp ressions m u st be of the sam e d ata typ e. If an exp ression
seeks to com bine a gau ge typ e variable w ith an integer typ e variable,
an error resu lts. mib.coerce is p rovid ed to coerce MIB variable to a
sp ecific typ e so they can be u sed in exp ressions.
MIB expression guru (ovexprguru)
Introd u ced in N N M 6.01, ovexprguru is an N N M com m and that is
u sed to p re-select the ap p rop riate MIB exp ression (mibExpr.conf) to u se
to grap h certain generic collections based on the collected object‟s
cap abilities. Its u se can be seen in the ARF files for N N M m enu bar
p erform ance and fau lt grap hs. The com m and takes the follow ing
generic keyw ord s and attem p ts to ap p ly the ap p rop riate m ibExp r from
the m ibExp r.conf file:
utilization
packetrate
averagepacketsize
thruput
errorrate
percenterrors
discardrate
datarate
percentdiscards
framerate
framedatarate
framecongestion
frameerrors
frameeligiblediscards
In the case of u tilization (and several others of the above a rgu m ents)
ovexprguru p olls the ifTyp e table and com p ares the resu lts to a table of
interface typ es that are hard -cod ed in the com m and . If the instance is a
fu ll d u p lex interface then the in and ou t u tilization are grap hed
sep arately. For a half d u p lex interface the in and ou t are com bined . For
m ore info, see the ovexprguru m an/ ref.
Defau lt Data collections based on MIB exp ression su ch as If%u til d o
not u se the MIB exp ression gu ru by d efau lt. In ord er rep lace an
existing collection for If%u til, follow these step s:
1.
Ad d a new MIB Expression in the $OV_CON F/ MibExpr.conf file, e.g.:
BetterIf%Util \
“Interface Utilization that applies the appropriate expression \ n\
Based on the ifType of the interface being monitored.”\
INDIRECT : utilization : COMBINED
2.
In the Data Collections and Threshold s , select the If%Util Object and
Ed it -> MIB Object -> Copy, then select the new expression from the
list of expressions.
3.
Suspend the If%Util collection and save. Run: ovstop snmpCollect
ovstart snmpCollect
150
Data Collection and Thresholds
N N M V7.51 Interm ed iate p atch 18 introd u ced an u nd ocu m ented
op tion to enable it to d u m p SN MP p ackets. The op tion is “ -d .”
Understanding and manipulating statistical thresholds
Statistical threshold ing w as introd u ced in N N M V6.2. This featu re u ses
stand ard d eviations on collected d ata that is exp orted to the Data
Warehou se. Previou sly, only fixed threshold s w ere available. When
configu ring a d ata collection object, “Fixed ”, “Both Statistical and
Fixed ”, or “Either Statistical or Fixed ” can be selected u nd er Threshold
Param eters.
Stand ard d eviation calcu lations requ ire the MIB variables be stored in
the snm p collect DB and exp orts of trend d a ta to d ataw arehou se are
enabled . The d efau lt tim e intervals for statistics are as follow s:
The d ata points are sorted into 3 buckets: Workhours (8am -5pm w eekd ays),
WeekDayOffHours (5pm -8am w eekd ays), and Weekend s.
These bu ckets can be changed u sing the follow ing configu ration file
w hich d oes not exist by d efau lt:
$OV_CON F/ statTimeRanges.conf
There is a m an/ ref p age for this file w hich d escribes how to configu re
it in d etail. The stats that are exp orted from the Data Warehou se to
bu ild the stand ard d eviations are stored in a flat file:
$OV_DB/ snmpCollect/snmpColStats.txt
snmpCollect commands
To d u m p configu ration d ata to $OV_LOG/ snm p Col.trace, ru n:
snmpCollect -S
To d elete snm p Collect d ata old er than 90 d ays (2160 hou rs):
ovcoltosql -N -D 2160
To rem ove all collected d ata for the snmpCollect d atabase:
rm –r $OV_DB/snmpCollect/
pingResponseTime, pingPercentRetry threshold events
These threshold events com e from d efau lt d ata collect ions that ship
w ith som e versions of N N M like 6.x (bu t not 7). They ind icate p otential
151
Fognet’s Field Guide to OpenView NNM
p roblem s w ith netmon p olling or w ith SN MP d ata collections. They can
be tu rned off or configu red from "Data Collections and Threshold s"
from the "Op tions" m enu .
These are actu ally "external" MIB exp ressions, and that m eans they are
MIB exp ressions that are d efined externally to the m ibExp r.conf file, so
are not exp osed to u sers. See the m an/ ref for m ibExp r.conf for m ore
inform ation.
D ata collection and device naming/IP address issues
The snmpCollect d aem on stores d ata by IP ad d ress. When xnmgraph is
invoked to read the d atabase, it p erform s a looku p to d eterm ine w hich
IP ad d ress to u se to recall d ata. There are several cases w here xnmgraph
fails to extract the ap p rop riate d ata from the snmpCollect d b. These
exam p les inclu d e:
Targets that use round -robin DN S
Targets that use H SRP
Cluster nod es
Targets that have been renam ed or re-IP ad d ressed
The d ata for the aged -ou t IP ad d resses can be retrieved by u sing the
exp licit IP ad d ress. A good tip is to u se loop back for m anagem ent and
rep orting for rou ters (see p age 26.)
OpenView specifics reserved for custom events
The stand ard threshold and rearm events are d esigned to be cop ied to
create cu stom events that p rovid e m ore m eaning and facilitate cu stom
actions. The follow ing sp ecific event id entifiers have been reserved for
u se for this p u rp ose:
1001-9999 odd
1002-10000 even
For threshold crossed or error events.
Custom rearm or return to norm al events
D ata collection and high speed links
Most SN MP stand ard MIB cou nters are 32-bit. A DS3 line at fu ll
band w id th cau ses a cou nter like ifInOctets to w rap in abou t 12
m inu tes. An FDDI line w rap s in less than 6 m inu tes. With a 64-bit
cou nter, how ever, a 1 tbs (terabits p er second ) link w ou ld take nearly 5
years to w rap at fu ll band w id th.
A cou nter w rap d u ring a p olling interval resu lts in an ignored valu e in
collected d ata. One w ay to im p rove grap h accu racy in this case is to
152
Data Collection and Thresholds
p oll high sp eed links m ore frequ ently to red u ce the nu m ber of ignored
d ata p oints, bu t this can ver y qu ickly cau se ad verse p erform ance
p roblem s (see p age 148). 64-bit cou nters w ere introd u ced along w ith
SN MPv2 to resolve these issu es and thu s are only fou nd in agents that
su p p ort SN MPv2 or greater.
The 64-bit cou nters that corresp ond to the stand ard interface table MIB
can be fou nd in the ifMib table, and echo their 32-bit cou nterp art‟s
nam e save for “H C” for H igh Cap acity inserted into their nam es, so for
exam p le: ifH CInOctets is fou nd at 1.3.6.1.2.1.31.1.1.1.6. See the su p p ort
m atrix on p age 296 for inform ation on N N M‟s su p p ort for variou s
versions of SN MP.
Exhausted 32-bit counters w hen 64-bit is supported
If the MIB su p p orts 64-bit cou nters, RFC 2233 (ifMib) m and ates
su p p ort of the corresp ond ing 32-bit cou nters for backw ard
com p atibility and states that these cou nters d o N OT reset betw een
32
64
2 - 1 and 2 - 1. H ere is the actu al verbiage:
When 64-bit counters are in use, the 32-bit counters
must still be available. They will report the low 32bits of the associated 64-bit count (e.g., ifInOctets
will
report
the
least
significant
32
bits
of
ifHCInOctets).
Most vend ors that su p p ly H C cou nters com p ly w ith this RFC. To test
this, p oll both the 32-bit cou nters and the 64-bit cou nters (e.g.
ifInOctets and ifH CInOctets, etc.) See if the 32-bit cou nters are
increm enting w hen the valu es for the 64-bit cou nters are greater than
32
2 - 1. If the 32-bit cou nters are not increm enting that cou ld exp lain
any u nd esirable resu lts. Mod ify the collections to u se the H C MIB
variables. For those system s in the collection that d o not su p p ort the
H C cou nters, sep arate them ou t and ap p ly the trad itional collections.
N ote that ovexprguru is able to test for 64 vs. 32-bit cou nter su p p ort.
Printing graphs (UN IX)
The d efau lt ou tp u t form at for grap hs is xwd. To convert this ou tp u t
into m ore fam iliar im age form ats, u se the conversion u tility at
$OV_CON TRIB/ N N M/ Im ageMagik. To p rint to an attached or
netw ork p rinter, for exam p le, a color LaserJet, com m ent ou t the d efau lt
d ata for the .p rintcom m and listed in $APP_DEFS/ XN m grap h and
rep lace w ith:
*.printcommand: xpr -device pjetxl -rv -header
"$OVTITLE" | lp -d <colourPrinterName>
153
Fognet’s Field Guide to OpenView NNM
For m ore inform ation of m od ifying ap p -d efau lts settings, see p age 9.
Insufficient memory w hen launching graphs (Solaris)
This sp ecific error m essage relates to m axMallocPercent resou rce and
occu rs m ostly on Solaris 2.7 servers. The error is cau sed by a too sm all
of a stack size, w hich can been seen w ith the com m and : ULIMIT –a.
To resolve, ru n: ULIMIT –s unlimited
Exporting graphs to other image formats (UN IX)
N N M grap hs can be saved by d efau lt in xw d form at. An excellent (and
free) u tility to convert from xw d to a variety of p op u lar form ats is
netp bm w hich can be fou nd at:
http:/ / netpbm .sourceforge.net/
D ata collections on VLAN s
SPAN or p ort m irroring can be u sed to m onitor VLAN traffic.
Typ ically, a w hole VLAN is m irrored to a single p ort to be m onitored .
Try not to oversu bscribe the d estination p ort, i.e., m irroring a 1000MB
VLAN to a 100m b p ort.
xnmgraph command examples
All N N M‟s m enu -bar p erform ance, fau lt and configu ration grap hs are
d irect invocations of the xnmgraph com m and line u tility. The
com m and invocations can be seen in the $OV_REGISTRATION / C
d irectory. For m ore info on ARF, see p age 6. Registration files for
grap hs p rod u ced u sing the MIB Ap p lication Bu ild er are p laced in the
ovm ib su bd irectory.
Grap h serial interface u tilization u sing d u p lex MIB exp ressions (“2” is
the ifInd ex of the target interface):
xnmgraph -title "Duplex Interface Utilization" -mib
IfInFDplxUtilization::2::::::,IfOutFDplxUtilization::2::
:::: <target>
<target> is the ip ad d ress of the d evice and the '2' in betw een all of the
colons is the ifInd ex of the interface to grap h. To grap h Cisco p ort bit
rates, inclu d ing real tim e and collected d ata (+brow se):
xnmgraph +browse –title ―Cisco Port Bit Rates‖
.1.3.6.1.4.9.2.1.1.1.6:label:3::::::
.1.3.6.1.4.9.2.2.1.1.8:label:3:::::: <target>
154
–mib
Data Collection and Thresholds
SN MP data collection terminology
PD U. Packaged Data Unit. A term for a m essage of a given protocol com prising
payload and protocol-specific control inform ation, typically contained in a head er.
PDUs exist betw een the OSI layers of protocols, and are not specific to any particular
layer. PDU is often used incorrectly to d escribe an SN MP Message.
Packet. A d iscreet u nit of data sent across a packet sw itching network, e.g. IP, Fram e
Relay, etc. Per RFC 1594: "‟Packet‟ is a generic term used to d escribe unit of data at all
levels of the protocol stack, but it is m ost correctly used to d escribe application data
units.” Packets contain data and a head er containing an ID num ber, source and
d estination ad d resses, and error-control data and can be of fixed or variable size. Used
interchangeably w ith datagram .
D atagram. Per RFC 1594, "a self-contained , ind epend ent entity of d ata carrying
sufficient inform ation to be routed from the source to the d estination com puter
w ithout reliance on earlier exchanges betw een this source and d estination com puter
and the transporting network." The d ifferentiator betw een a dat agram and a packet
centers on this concept of self-sufficiency. Because of this, there is an im plication that
d atagram s are UDP (connection -less) vs. TCP, w hich is connection -oriented , but this is
not always true.
MTU. The Maxim um Transm ission Unit is the largest packet that a given netw ork
m ed ium can carry. Ethernet, for exam ple, has a fixed MTU of 1500 bytes, ATM has a
fixed MTU of 48 bytes, and PPP has a negotiated MTU that is usually betw een 500 and
2000 bytes.
Octet. Eight contiguous bits of data. A byte is often 8 bits of data, but not alw ays. An
Octet is always 8 bits of data.
Frame. In generic term s, a fram e is a basic logical unit in w hich bit-oriented data is
transm itted . An Ethernet Fram e is equivalent to a packet. A Frame Relay fram e is a
d ata unit containing a start-of-fram e (SOF) d elim iter, head er, payload , cyclic
red und ancy check (CRC), and an end -of-fram e (EOF) d elim iter. The payload can be 02112 bytes, and the CRC is 4 bytes.
Fragment. A piece of a packet, also called a runt. When a router is forw ard ing an IP
packet to a network that has a m axim um packet size sm aller than the transm itted
packet size, it is forced to break up that packet into m ultiple fragm ents. These
fragm ents should be reassem bled by the IP layer at the d estination host.
Payload. The Payload is the part that represents application inform ation and
application overhead information in a set of d ata being processed or transported.
System Insight (Compaq Insight) Manager MIBs
The CIM/ SIM MIBs are load ed by installing the SIM/ N N M
integration p ackage (URL for this p ackage is on p age 300). To
m anu ally load the MIBs w hen not u sing the integration p ackage, load
in the follow ing ord er:
1. cpqhost.m ib
5. cpqstd eq.m ib
155
Fognet’s Field Guide to OpenView NNM
2. cpqhealth.m ib
3. cpqsinfo.m ib
4. cpqsrvm n.m ib
6. cpqstsys.m ib
7. cpqthrsh.m ib
8. cpqups.m ib
SN MP data collection vs. RMON
RMON (RFC1757) and RMON -II (RFC2021) are u sed interchangeably
below . The d ifference betw een SN MP and RMON is that the “M” in
SN MP stand s for m anagem ent w hereas the “M” in RMON stand s for
m onitoring. RMON d ata is aggregated by RMON p robes, or agents,
and these are im p lem ented as SN MP MIBs. While RMON p rovid es a
m u ch m ore efficient w ay to m onitor traffic at or nearer to the sou rce,
for the m ost p art, vend ors are m oving aw ay from su p p orting RMON
and relying m ore on SN MP and p acket cap tu ring tools to p rovid e the
m onitoring fu nctions. N ote that N N M, how ever, d oes not p rovid e any
p acket cap tu ring cap abilities ou t of the box.
There are d ifferent "layers" of RMON , and d ifferent cap abilities that a
nod e m ay im p lem ent. Most RMON im p lem entations d o not conform
to RFC stand ard s, relying on p rop rietary extensions to the stand ard
MIBs to p rovid e better m onitoring. Exam p les of RMON u se range
from gathering sim p le statistics abou t the netw ork like u tilization and
collisions, to actu ally cap tu ring p ackets w hich are then retrieved
rem otely. There is also SMON , w hich is RMON for sw itched netw orks
created by Lu cent, bu t few vend ors other than Lu cent have
im p lem ented it. In fact, few vend ors w ho su p p ort RMON bother to
fu lly im p lem ent the stand ard p arts of it. They often only inclu d e the
first few RMON grou p s and leave the fu ll p acket cap tu ring cap abilities
to their p rop rietary extensions. That said , som e u sefu l trou bleshooting
d ata can be m ined from RMON . Look into the statistics grou p to
d eterm ine if abnorm al traffic is flow ing over a p articu lar p ort . Details
m ay inclu d e fram e+ size d istribu tions, u nicast/ m u lticast/ broad cast
and fram ing error typ es su ch as ru nts and giants.
Calculate epoch time for snmpColD ump, etc.
Tim e stam p s for N N M d atabase record s for snmpCollect d ata and for
events are kep t as ep och tim e (second s since Janu ary 1, 1970). Place
the ep och tim e to translate w here is says “c” below :
%OV_BIN%\Perl\bin\perl -e ―print scalar localtime(c)‖
$OV_BIN/Perl/bin/perl -e ‗print scalar localtime(c)‘
156
14. Notifications
N otifications from N N M are generated as action callbacks d efined
w ithin the event su bsystem . There are no actions associated w ith SN MP
Trap s, so all actions are d efined from the m anager. N ote that m ost N N M
events are generated by N N M itself as the resu lts of statu s p olls or other
anom alies d etected by N N M‟s d iscovery and p olling engines. The
ovactiond d aem on is resp onsible for ru nning actions assigned to ev ents,
and actions are configu red in the event configu ration GUI (xnmtrap).
Actions associated w ith events are called „action callbacks‟.
Configuring an action callback
Follow these general step s w hen setting u p an au tom atic action based
on an N N M event:
1. Configure and test the script/ tool outsid e of NN M from the com mand line
(UN IX: By d efault, the actions ru n w ith UID and GID of user bin)
2. Select useful variable bind ings to pass. The variable bind ings are often
listed in the event d escription. For m ost OpenView enterprise events, $2
represents the event source. Ad d itional variables are listed below u nd er
Selected Special Variables
3. Configure trusted Cm d s.conf (see below )
4. Optional step. Test the com m and using the OpenView event
OV_application_alert in conjunction w ith the
$OV_CON TRIB/ NN M/ send Msg/ send Msg.ovpl script
Using trustedCmds.conf file
In N N M 6.2+, only com m and s listed in a file in the follow ing d irectory
can be execu ted by ovactiond:
$OV_CON F/ trusted Cm d s.conf/
Changes to files in tru sted Cm d s.conf m u st be signaled to ovactiond by
ru nning the xnmevents –event com m and from the com m and line.
tru sted Cm d s.conf is d ocu m ented in the m an/ ref p age for ovactiond.
157
Fognet’s Field Guide to OpenView NNM
Any file p laced in that d irectory is p arsed for valid com m and nam es
(w ith absolu te p ath) and aliases, for exam p le, su p p ose a file called
ringbell in $OV_CON F/ trusted Cm d s.conf contains:
ringBell=/opt/OV/contrib/NNM/ringBell/ringBell.ovpl
In the "Com m and for Autom atic Action" field (action callback), use “ringBell”.
Overrid e the tru sted com m and s featu re by creating a file nam ed
ALLOW_ALL in the $OV_CON F/ tru sted Cm d s.conf d irectory. If this
file, w hich sh ou ld be em p ty, is fou nd then ovactiond d oes not check
w hether a com m and being execu ted is tru sted .
General action callback usage considerations
Exit cod es. ovactiond listens for the exit cod es retu rned from scrip ts or
p rogram s and rep orts a failed action if it d etects a non -zero exit cod e.
UN IX UID for ovactiond: ovactiond execu tes com m and s as u ser bin, and
thu s is not environm ent or p ath aw are. To change the u ser that
ovactiond execu tes as, ad d the –u <user> LRF sw itch in ovactiond .lrf.
See p age 5 for the LRF p roced u re. UN IX UID and ping: If the action
callback execu tes (or lau nches a scrip t that execu tes) p ing, the
com m and m ay p rod u ce a failu re to op en socket. ping m u st be
execu ted as the root u ser becau se it op ens a raw socket and only root
on UN IX can op en raw sockets. Use the –u root op tion d iscu ssed
above.
Scalability: When scrip ting or calling p rogram s, it m ay be necessary to
consid er issu es w h ich m ay arise from m u ltip le su ccessive calls to
ovactiond. Consid er configu ring action s to fork, sp aw n or exec so the
child ren d o the w ork and the p arents exit, allow ing ovactiond to lau nch
the next bu ffered p rogram calls. Sem ap hore locking m ay also be a
u sefu l w ay to d eal w ith overru n issu es associated w ith event flood s. In
this case, the first p rocess gets the lock and d oes the w ork. The second
p rocess fails to get the lock becau se som eone else is alread y hand ling
the issu e, and exits. Also consid er event correlation. The correlation to
hand le d u p licate events can be ad d ed very easily u sing the d ed u p .conf
(see p age 125), and p revent m u ltip le p ages, for exam p le. For som e
correlations like PairWise (p age 124), release of the action m ay be
d elayed p end ing the receip t of an event (w hich m ay clear the initial
cond ition). See p age 159 below for som e m ore notes on interactions of
ECS w ith notifications.
158
Notifications
Metacharacters in action callbacks
Action callbacks d o not su p p ort Regu lar Exp ressions. The caret (^ ), for
exam p le, is u sed to qu ote. The list of m etacharacters is:
;
&
|
<
>
new-line
In ad d ition, any com m and su bstitu tion in the varbind s is also qu oted .
Any string enclosed in ' ' or $() is consid ered a com m and su bstitu tion.
Selected special variables in action callbacks
This is a p artial list of som e of the m ore u sefu l variables:
$#
$*
$n
$x
$X
$V
$ar
$c
$s
$$
$E
$A
$aA
$G
$S
$T
The num ber of attributes in the event.
All attributes as “seq nam e (type): value” strings.
nth attribute as a value string. Must be 1 to 99.
Date event received using the local d ate representation.
Tim e event received using the local tim e representation.
Type: SN MPv1, SN MPv2C, CMIP, or GEN ERIC.
The im plied source as an IP add ress.
The category the event belongs in.
The severity of the event.
The $ character.
The trap enterprise as a text string if possible.
The trap agent ad d ress as d efined in the trap PDU.
Sam e as $A except the source as an IP ad d ress.
The trap's generic-trap num ber.
The trap's specific-trap num ber.
sysUptim e in hund red ths of a second since agent up
These characters can be sp ecified d irectly in the action callback:
a
b
f
n
r
t
v
\
ooo
xhh
Alert (bell) character
Backspace
Form feed
N ew line
Carriage return
H orizontal tab
Vertical tab
Shell escape (\ \ ind icates Wind ow s pathnam es.)
Octal num ber in range 000-177
H ex num ber in range x00-xFF
N otification interactions w ith event correlation
After N N M 6.31, certain statu s events m ay be “held ” by the N od eIf
and / or Pairw ise event correlations p end ing related events. Au tom atic
159
Fognet’s Field Guide to OpenView NNM
actions are held as w ell, and the related events cou ld cau se the child
event and its action to be d iscard ed .
All p rior versions of N N M alw ays released au tom atic actions
im m ed iately. Follow ing that, certain ECS circu its m ay affect the tim ing
of the release of au tom atic actions. In p articu lar, the PairWise and
N od eIf correlations m ay or m ay not affect the tim ing of the release of
au tom atic actions, d ep end ing on the version of N N M. To force
au tom atic actions to ru n im m ed iately, regard less of event correlation
settings affecting actions, ad d the –f RAW flag to the ovactiond d aem on
via the LRF p rocess.
ov act iond LRF settings
See p age 5 for general instru ctions on m od ifying LRF flags.
-u username
-t
-v
-l
(UN IX only) set UID and GID for ovactiond
N ote: default is bin:bin
Trace execution of ovactiond
More verbose logging
Use logfile other than $OV_LOG/ ovactiond .log
-w maxwait
Second s to wait for non -zero exit cod e before
giving up and killing execution. 0 ind icates
never give up; 300 is d efault
-f flowtype
RAW, CORR, or ALL; see Interactions w ith
Event Correlation (above)
-s maxlog
Maxim um size of logfile; d efaults to 500K
Limiting Action callbacks to a subset of objects
To set u p an action that ru n s only a su bset of event sou rces, m ake a
cop y of the target event, give it a u niqu e nam e, and sp ecify sou r ces
u sing “ad d from m ap .” See p age 95 for m ore basics on m anu al event
configu ration.
“Find ” is a p ow erfu l tool for bu ild ing lists of nod e sou rces based on
com m on attribu tes. Once a set of objects is highlighted on the m ap as
the resu lt of a “find ” op eration, u se “View - Select H ighlighted ” to
select objects across m u ltip le su bm ap s, then “ad d from m ap ” w ithin
the Event Configu ration GUI.
Selection nam es can be w ild card ed in the event sou rce sp ecification.
For exam p le, *sou p * w ou ld m atch selections p easou p and sou p y. IP
160
Notifications
Ad d resses can also be u sed as event sou rces, and * or - w ild card s can
be sp ecified as w ell, for exam p le: 10.10.0.* or 10.10.0.2-252.
See the section below on u sing an external file to sp ecify
External File to specify action callback sources
To u se an external list of sou rces, enter the fu ll p ath to a text file
containing valid selection nam es or IP ad d resses, one p er line, in the
sou rces area of the Event Configu ration GUI. Use xnmevents –event to
force a re-read of the file after m aking u p d ates.
Wild card s cannot be u sed in this external file for either selection nam es
or IP Ad d resses. Wild card s can, how ever, be u sed d irectly in the event
sou rce sp ecification (see section above).
Using Perl for actions callbacks
N N M is ship p ed w ith an em bed d ed version of Perl located at
$OV_BIN / Perl/ bin/ p erl. Taking ad vantage of this is fine for relatively
sim p le scrip ts, bu t it is inad equ ate for scrip ts that call for ad d itional
Perl Mod u les becau se N N M‟s Perl d oesn‟t incorp orate all the libraries
necessary for extensibility. Install or u se a fu ll-blow n Perl d istribu tion
if calling p erl m od u les in scrip ts. For Wind ow s p latform s, ActiveState
Perl com es highly recom m end ed . To u se N N M‟s em bed d ed Perl on
any p latform , sim p ly u se the “.ovp l” file extension for the scrip t. On
UN IX, call Perl at the top of the scrip t u sing:
#!/opt/OV/bin/Perl/bin/perl
When m aking action callbacks to Perl scrip ts in UN IX, enclose the
argu m ents to be p assed in d ou ble qu otes, for exam p le:
/opt/OV/bin/myperl.ovpl "$x $X $r $R $*"
When m aking action callbacks to Perl scrip ts in Wind ow s, either alias
Perl u sing tru sted Cm d s.conf, u se the registered .ovp l if u sing the
N N M-em bed d ed version of Perl, or call the fu ll p ath to an alternate
Perl execu table, as in:
cmd /c start /min c:\\perl\\bin\\perl.exe c:\\perl.pl $2
Action callbacks on Window s platforms
The m ost com m on issu e w ith calling scrip ts on Wind ow s p latform s is
d ifficu lties in interp reting p ath nam es w ith em bed d ed sp aces and in
hand ling sp ecial characters. Use d ou ble qu otes in this case as in the
follow ing action callback exam p le:
161
Fognet’s Field Guide to OpenView NNM
d:\\‖Program Files‖\\‖HP OpenView‖\\bin\\perl\\bin\\perl
d:\\‖Program Files‖\\‖HP OpenView‖\\scripts\\pager.ovpl $r
The sim p lest w ay to avoid su ch trou ble is to u se aliases for com m and s
via tru sted Cm d s.conf (see the ovactiond m an/ ref). OVSH ELL and
OVH IDESH ELL are sp ecial keyw ord s available u nd er N N M on
Wind ow s to d isp lay action resu lts in a new w ind ow or to su p p ress the
p op u p of a com m and shell. OVSH ELL is the d efau lt. These keyw ord s
also w ork w ith aliases d efined in tru sted Cm d s.conf. Exam p le:
OVSHELL notepad file.txt (same as ―notepad file.txt‖)
OVHIDESHELL cmd.exe /s file.bat
OVHIDESHELL telalert –i phil –m ―$2 down‖
In the first exam p le, the p op u p w ind ow is forced . In the second
exam p le, an exp licit callback is issu ed that w ou ld requ ire the
ALLOW_ALL file exists in tru sted Cm d s.conf. In the third exam p le, the
fu ll p ath to the telalert execu table is d efined in tru sted Cm d s.conf and
aliased to “telalert.” The equ ivalent to OVH IDESH ELL can be
accom p lished w ith:
cmd /c start /min command c:\\ bmail -s [email protected] –t
[email protected] -f [email protected] -a "$2 down‖ -b "Body text"
Or, for fans of Wind ow Scrip ting H ost, p op -u p s can be su p p ressed
w ith this exam p le:
Set WshShell = CreateObject("WScript.Shell") RetVal =
WshShell.Run("CMD.EXE /C dir/w",0,TRUE)
Som etim es, it is d esirable to keep the com m and w ind ow u p , as in
w hen inp u t is requ ired by the calling p rogram , for exam p le:
start /wait c:\\shutdown\\shutdown.exe
There is a p roblem w ith u sing p ip es (| ) or red irects (>) in action
callbacks on Wind ow s. These are su p p orted on the Wind ow s
com m and line, bu t ju st d on‟t w ork in N N M action callbacks.
Email notifications using native tools (Window s)
IIS is installed w ith N N M. IIS has SMTP cap abilities, w hich ca n be
configu red from the IIS p rop erties p age. Once configu red , IIS attem p ts
to d eliver files w ith the p rop er form at that are d rop p ed in the
follow ing d irectory via SMTP:
C:\ Inetpub\ mailroot\ Pickup
162
Notifications
A Perl scrip t snip p et that can be u sed as an action callback to send an
em ail to a p aging service u sing Microsoft‟s IIS‟s bu ilt-in SMTP server is
show n here:
$node = $ARGV[0]; # $r.
open OUT, ">c:\\page.txt";
print OUT "From: openview\@domain.com\n";
print OUT "To: 12345678910\@archwireless.net\n";
print OUT "Subject: Node $node Down\n";
print OUT "\n";
print OUT "Node $node Down\n";
close OUT;
`copy c:\\page.txt c:\\inetpub\\mailroot\\pickup`;
`del c:\\page.txt`;
exit;
MAPISEN D is p art of the Exchange Resou rce Kit. It is on the TechN et
CDs. To configu re this, install Ou tlook on the m onitoring server and
configu re a p rofile u sing the service accou nt that is being u sed for
N N M. H ere is an exam p le MAPISEN D com m and action callback:
MAPISEND.EXE -u <outlook profile name> -p <service account
password> -r [email protected] -s $2 Down -m ―Node $2 is down‖
Email notifications using native tools (UN IX)
mailx and elm are native to both H P-UX and Solaris OS. Both u se
very sim ilar syntax w hen invoking from the com m and line. In the
follow ing exam p les, the elm and mailx com m and s cou ld be
interchanged :
echo "$r Up" | mailx -s "$r Up" [email protected]
elm -s "$2 down" [email protected] < /tmp/body.txt
sendmail can also be u sed , for exam p le:
echo 'Subject: Openview Trap object $r for event $N $1
$3\n' | /usr/lib/sendmail '[email protected] [email protected]‘
And yet another w ay for Solaris system s, d em onstrating how to p ass
nu ll as the file hand le w here one is requ ired :
/usr/ucb/mail -s "$2 down" [email protected] </dev/null
Email notifications and D N S issues (UN IX)
Som e enterp rise em ail servers d o not forw ard em ail along u nless the
send ing system form ats the originating ad d ress as an FQDN . The
p reffered m ethod to d o this is by p rop erly configu ring the
/ etc/ resolv.conf file on the m anagem ent server w ith a line that
contains: “search d om ain.com ”. Another m ethod w hich m ay
163
Fognet’s Field Guide to OpenView NNM
accom p lish this, if for exam p le N IS is being u sed , is to set the “Dj”
setting in the sendmail.cf file.
Audio notifications (UN IX)
Use $OV_CONTRIB/NNM/ringBell/ringBell.ovpl to ring the bell on
the N N M server. To m od ify the volu m e and d u ration of the sou nd ,
ed it the $APP_DEFS/ XN m events file and find section on ringing the
bell. Rem ove the "!" com m ent, set as d esired (try the m axim u m first bellp ercent 100).
To ring the bell of any rem ote system that accep ts X d isp lay red irects,
inclu d ing PC‟s ru nning X w ind ow s softw are, first create a scrip t
“beep .sh” that looks like this:
#!/bin/sh
echo "This window beeps then closes in 1 second"
echo "$1 down" #args passed in order from callback
echo ^G #Use real control-G (in vi): CRTL-V CRTL-G
sleep 1
N ext ad d the action callback like this:
/usr/bin/X11/xterm -display <IPAddr>:0 -e /beep.sh
To p lay an au d io file (Solaris only), try:
cat crash.au > /dev/audio
It m ay be requ ired to change the p erm issions for / d ev/ sou nd to 666 in
the / etc/ logind evp erm s file for this to w ork.
To enable PC‟s ru nning Exceed X-w ind ow s em u lation, ed it w in.ini and
ad d the follow ing:
[exceed]
DefaultSystemBeep=1
Sound notifications (Window s)
Install Wind ow s Med ia Player, or som e other d efau lt WAV file p layer
that can be invoked from the com m and line. For the action callback,
u se:
mplay32 c:\\<path>\\file.wav
Example notification setups
Exam p le u sing BLAT (Wind ow s):
1. Create $OV_CON F\ trustedCm d s.conf\ testmail containing:
testmail=c:\\testmail.bat
164
Notifications
2. Run xnmevents –event from a com m and shell
3. C:\ testm ail.bat contains:
C:\blat - -subject "%1 is down‖ -to [email protected] body "%1 is down at %2 on %3 "
4. Action callback for OpenView OV_N od e_Dow n event:
OVHIDESHELL testmail $2 $X $x
Exam p le u sing Telalert w ith ALLOW_ALL (Wind ow s):
1. Create em pty file:
$OV_CON F\ trusted Cm d s.conf\ ALLOW_ALL
2. Run: xnmevents –event from a com m and shell
3. Action callback for OpenView OV_N od e_Dow n event:
cmd /c start /min d:\\vytek\\telalertc –i mypager –m ―$2
down‖
Exam p le Perl scrip t that sp aw ns so ovactiond retu rns:
1. Create em pty file:
$OV_CON F\ trusted Cm d s.conf\ ALLOW_ALL
2. Run: xnmevents –event from a com m and shell
3. Action callback for OpenView OV_N od e_Dow n event:
/ usr/bin/perl do_my_thing.pl $A $1 &
Popular notification and paging tools
The follow ing p rod u cts are often m entioned by N N M end -u sers:
Com m ercial Prod u cts:
Vend or
Prod uct
Calam p
Sem otus
Invoq
iTechTool
Inventive
PageGate
Spatch
Sysm an
Telalert
H iplink
Alarm point
EtherPage
Pow erPage
N otePage
Spatch
SMS Server
URL
w w w.calam p.com
w w w.sem otus.com
w w w.alarm point.com
w w w.ppt.com
inventivelabs.com
notepager.net
w w w.spatch.com
w w w.sysm an.no
Platform (s)
Unix/ Window s
Unix/ Window s
Unix/ Window s
Unix
Window s
Window s
Unix/ Window s
Wind ow s
Freew are/ Sharew are/ GPL/ Op en Sou rce:
Prod uct
Send page
send page
Blat
Bm ail
qpage
Postie
URL
send page.cpoint.net
w w w.send page.org
w w w.blat.net
beyond logic.org/ solutions
w w w.qpage.org
infrad ig.com
Platform (s)
Unix
Perl (Both)
Wind ow s
Wind ow s
Unix
Wind ow s
165
Fognet’s Field Guide to OpenView NNM
Usefu l link for Tap Mod em Ph one nu m bers:
http:/ / w w w .notepager.net/ tap -phone-num bers-a.htm
Scrolling m essage d isp lays that accep t p ager inp u t:
http:/ / w w w .am s-i.com
Configuring an attached modem on UN IX servers
While som ew hat of a challenge, setting u p a d irectly -attached m od em
for p ager notifications can increase the reliability of notification
system s that m ay rely on the services (em ail) they are m onitoring as a
transp ort for those notifications, thu s elim inating the p ossibility that
p ages are m issed if a d ep end ent service (su ch as a netw ork connection
to the em ail server) fails. cu (call u nix) can be u sed to in itiate p ages to
d irectly-attached m od em s. To u se cu, configu re u u cp to the p ort the
m od em is attached to as follow s:
Solaris:
Set serial port on Solaris / d ev/ cua/ a for a mod em to d ial up:
chmod +x /devices/sbus@1f,0/zs@f,1100000:a,cu
H P-UX:
SAM – peripherals – term inals & m od em s, m od em
Ed it / etc/ uucp / Devices
Ad d the follow ing line to the end of the file:
Direct cul0p2 - 19200 direct
Note: The actual cu d evice correspond s to the tty returned by SAM
The follow ing com m and can be u sed on Solar is to invoke kermit to
send a p age to a locally attached m od em :
kermit -l /dev/cua/a -m hayes -b 9600 -C \"set dial
timeout 15,dial {$pager_number,,,$message_string
#},hangup,exit\"
$OV_CONTRIB/NNM/beep95_lx/beep95_lx.sh is a front-end to the cu
scrip t, bu t it w as rem oved from later versions of N N M d istribu tions. A
cop y can be fou nd in som e search engines. A Perl scrip t can be
constru cted to interact w ith cu u sing the op en2 Perl Mod u le, for
exam p le in the follow ing snip p et:
use IPC::Open2;
use Symbol;
$WTR = gensym();
$RDR = gensym();
$pid = open2($RDR, $WTR, "cu -s 9600 -b 7 -t -e -l
/dev/cua/b");
(code goes here to dial and communicate)
print $WTR "~."; # terminate cu
close ($WTR);
close ($RDR);
166
15. Fault Analysis Tools
This section focu ses on tools w hich are p art of N N M (and som e w hich are
not) that are u sed to trou bleshoot netw ork fau lts.
Problem diagnostics (PD )
PD is inclu d ed w ith N N M AE v7.0+ and is enabled along w ith ET via
setu p ExtTop o.ovp l. On initialization, it installs a p robe on the N N M
server. Ad d itional p robes can be installed from one of the files
p robeH P.tra, p robeSUN .tar or p robeWIN .zip , fou nd in the
$OV_Install_Dir/ p d AE/ bin d irectory.
V7.53 inclu d ed im p roved
d ocu m entation for PD in the Gu id e to Using Extend ed Top ology u ser
m anu al.
Problem Diagnostics p robes netw ork flow s betw een critical d evices,
au tom atically base lining p ath p erform ance. It send s events if
p erform ance d eviates from the norm , and d etects p ath blackou ts,
brow nou ts, rou ting loop s, flap p ing p aths and other instability. PD u ses
tracerou te to d eterm ine Level 3 p aths and u ses ET top ology d ata to
p lace Level 2 d evices in p aths.
To start/ stop the p robe:
UN IX:
$OV_MAIN_PATH/pdAE/bin/pdcentral.sh –start|stop
DO NOT use the kill -9 com mand on the Java process! Irrecoverable data
corruption may occur.
Wind ow s:
From the Services applet, select NetPath and click Start (or Stop).
DO NOT use the Window s Task Manager to term inate the Java process! Irrecoverable
d ata corruption may occur.
To d isable a p robe:
UN IX:
Move the file from the rc3.d d irectory.
Wind ow s:
167
Fognet’s Field Guide to OpenView NNM
Use the Services applet to stop the N etPath service, and run:
pdcentral.bat –uninstall
To u ninstall p robe and server:
UN IX:
pdpuninstall.sh
Wind ow s: pdpuninstall.vbs
To link the p robe to m u ltip le servers:
Ed it npprobe.conf on the probe system , m od ify existing lines, then stop and restart the
probe. A probe uses the sam e port to talk to all servers.
Logs (located in $OV_MAIN _PATH / p d AE/ logs):
pd .log
npprobe.log
m essages from the PD Central application
m essages from the probe on this system
Trou bleshooting PD:
 If the GUI applet is not w orking check java console w ithin brow ser for exceptions
 If PD Central d oesn‟t start via ovstart, try using ovstop pd, then running PD
m anually via pdcentral.sh –start or pdcentral.bat –start.
 Use “<DEBUG>true</ DEBUG>” in the pdconfig.xm l file to generate d ebug output
in the pd.log file. This option should only be used briefly because it can generate
large amounts of data
 Use pdcentral.sh –dbmgr (or pdcentral.bat –dbmgr) to launch a UI that
allow s SQL queries on the PD d atabase
Smart path (access path view )
Sm art p ath is a contribu ted ap p lication ad d ed in V7.51. Sm art p ath has
tw o fu nctions: the first is to grap hically show the Layer 2 and Layer 3
access p ath betw een tw o nod es and the seco nd is to find a rou gu e
MAC ad d esss. The first fu nction show s u p as a new d ynam ic view
called Access Path view . The second fu nction is a com m and line tool.
Access p ath view is not enable by d efau lt.
Enabling and d isabling Sm artPath w ill stop and restart the ovas
p rocess To enable Sm artPath, ru n the follow ing com m and :
$OV_CON TRIB/ N N M/ Sm artPath/ ovAccessPath.ovpl –enable
The new Dynam ic View is available in the d rop d ow n m enu of N N M‟s
H om eBase. This view accep ts tw o end p oints as inp u t. Sm artPath
consu lts the N N M Extend ed Top ology and m akes a nu m ber of SN MP
qu eries w hich m ay take several m inu tes.
If m u ltip le MAC ad d resses are forw ard ed on the last p ort of the last
sw itch, an “Unknow n” d evice is ad d ed to the p ath. This hap p ens
w hen a hu b or L2 d evice has not been d iscovered by ET.
168
Fault Analysis Tools
Find rouge MAC addresses
To find the sw itch that a MAC ad d ress is connected to:
$OV_CON TRIB/ NN M/ Sm artPath/ ovAccessPath.ovpl –mac <MAC> –file <file>
$OV_CON TRIB/ NN M/ Sm artPath/ ovAccessPath.ovpl –mac <MAC> –router <ip>
<MAC> is the MAC Ad d ress in qu otes, for exam p le: “00 AA BB CC
DD EE”. If the -file p aram eter is u sed , the L2 d evices sp ecified in the
file are exam ined to find the MAC ad d ress. The file contents ar e
exp ected to be one line p er IP ad d ress. If the -rou ter p aram eter is
sp ecified , ET is qu eried for the L2 neighbors of the rou ter, w hich are
then exam ined to find the MAC ad d ress.
The ou tp u t is an XML d ocu m ent that show s w hich sw itches the MAC
is forw ard ed on as w ell as the board / p ort. For each sw itch, a flag is
inclu d ed that ind icates if m ore than one MAC is being forw ard ed
on the p ort. The sw itch p ort w hich only forw ard s the sp ecified MAC is
likely to be the sw itch that the MAC is w ired to.
The ping tools
The Fau lt-Ping m enu op tion u sed to call the native OS Ping u tility, bu t
after N N M 6.2, the natping w rap p er w as ad d ed to convert the p rivate
IP into the p u blic IP ad d ress w hen p inging into N AT environm ents.
This also changed the tool from u sing a continu ou s p ing to one that
only issu es 5 ICMP requ ests then stop s. This is controlled by the “ -n
5” op tion in the natping scrip t in $OV_BIN . Rem ove the op tion in
that file to restore continu ou s p ings. Also, to allow the u ser to stop the
p ing w hile it is ru nning, change the follow ing line:
"system ("$pingExe $PublicIP $pingArgs")"
to:
"exec ("$pingExe $PublicIP $pingArgs").
In ad d ition, ad d the –followOutput p aram eter to the xnmappmon
call to
natping
in the N N M-IP.tbl registration file in
$OV_REGISTRATION / C/ ip m ap . This allow s the end of the ou tp u t to
continu ou sly d isp lay so the u ser d oesn‟t have to scroll d ow n to see the
resu lts. See p age 6 for m ore info on m od ifying ARF files.
169
Fognet’s Field Guide to OpenView NNM
Isolating local netw ork issues
The follow ing com m and s can help isolate netw ork p erform ance issu es
related to the N N M server, its netw ork stack, and the locally -attached
N IC. Check for UDP-related socket overflow s:
netstat –p udp
Overflow s can ind icate SN MP traffic is overw helm ing the IP
This can be becau se of over aggressive d ata collections, or too
d evices being p olled . On UN IX, u se ndd to tu ne the netw ork
N ote this com m and rep laces the old er nettune com m and u sed
H P-UX.
stack.
m any
stack.
u nd er
rnetstat
rnetstat is an N N M-su p p lied SN MP-based version of the fam iliar
netstat com m and for p u lling basic netw ork stack d ata via SN MP.
rnetstat is also the front-end to som e of the configu ration, fau lt and
p erform ance m enu -bar item s in the N N M GUI. Som e of the follow ing
rnetstat sw itches are not d ocu m ented or su p p orted by H P:
List route table:
List connected and listening ports:
If nam es, ad d rs, m ask, net add rs:
If nam es, status, type, cap., alias:
rnetstat
rnetstat
rnetstat
rnetstat
–rn <target>
–a <target>
–In <target>
–o ifsum <target>
rnetstat –a is a very p ow erfu l tool for retrieving connected p ort
info, and w orks by cross-referencing the retu rned MAC ad d resses to
the $OV_CON F/ p hysAd d r.conf file. This file can be cu stom ized , bu t is
over-w ritten by N N M u p grad es, so take ap p rop riate p recau tions.
Som etim es u sers p refer not have the MAC ad d resses translated by the
rnetstat com m and . In this case, sim p ly com m ent ou t p articu lar
entries in the p hysAd d r.conf file or renam e the p hysAd d r.conf file
entirely.
D isplay port address mappings
There is a m enu -bar item u nd er tools-Port-Ad d ress Map p ing Table
w hich d isp lays p ort connections for all d evices connected to the
selected nod e. The URL for this tool is:
http:/ / <server>:3443/ OvCgi/ connectedN od es?nod e=<nod e>
…w here <nod e> is the FQDN of the target nod e and <server> is the
nam e of the N N M server. This tool is a front-end for the rnetstat
com m and d iscu ssed im m ed iately above.
170
Fault Analysis Tools
netcheck
netcheck is a hand y bu t p oorly d ocu m ented N N M tool that is the gu ts
behind the m enu item : Fau lt ->Test IP/ TCP/ SN MP. It can be issu ed
from the com m and line to test one or m ore of the three p rotocols ju st
m entioned . Tw o exam p les follow :
netcheck -e -o tcpPort=telnet <target>
netcheck -e -o tcpPort=21 <target>
N ote if the nam e of the service sp ecified , as in the first exam p le above,
the tool cross-references the service p ort w ith the services file.
mibtable
This com m and d u m p s table view s of SN MP MIB su btrees in text
form atted ou tp u t, and like rnetstat, is a very p ow erfu l and p op u lar
tool for N N M ad m inistrators. For exam p le, to d isp lay p ort connection
table entries:
mibtable -table ".1.3.6.1.2.1.4.22.1" -fields
"Index=1:3,PhysAddress=2:18,IPaddress=3:15" -node <node>
On UN IX system s, this exam p le p rod u ces mibtable ou tp u t that is
nicely form atted :
mibtable -table ".1.3.6.1.2.1.4.22.1" –fields
"IPAddress=3,Physaddress=2" -node somenode | nawk
'$0~/^[1..9]/ {printf ("%-15s %s-%s-%s-%s-%s%s\n",$1,$2,$3,$4,$5,$6,$7)}'
tracert and traceroute
These native com m and s, w hich are u sed in N N M‟s Fau lt m enu , show
the netw ork hop s a p acket takes to the d estination. The p ath need not
alw ays be the sam e, nor m ay be the retu rn p ath. Both com m and s are
native OS com m and s. N N M Wind ow s ship s w ith a cop y of
tracert.exe in %OV_BIN %, bu t it is no d ifferent that the native one.
Wind ow s tracert d iffers from UN IX traceroute in that it is lim ited
to u sing ICMP exclu sively (traceroute u ses UDP d atagram s or
op tionally, ICMP). Wind ow s also has native u tility called pathping
w hich p rovid es broad er p rotocol su p p ort than Wind ow s‟ tracert.
With N N M 7.0+, Problem Diagnosis is bu ilt into N N M. This tool
p rovid es m ore com p rehensive p athing and statu s d ata if PD p robes
are p rop erly configu red . See the beginning of this section for m ore on
Problem Diagnosis.
171
Fognet’s Field Guide to OpenView NNM
Packet sniffers
N N M d oes not contain a bu ilt-in tool for p acket sniffing. H ow ever,
som e N N M d aem on tracing facilities can p rod u ce d etailed p acket
d u m p s, for exam p le pmdmgr (see p age 131). There are som e com m ercial
and freely available p acket sniffers that are w id ely u sed to su p p lem ent
the N N M u sers trou bleshooting toolset. Follow ing is a short list of
selected sniffers that are u sed by som e N N M ad m inistrators.
N etw ork Monitor (netmon.exe) is a p acket sniffer p rovid ed by
Microsoft that is available in Microsoft Resou rce CDs. nettl/netfmt is
a UN IX native p acket d u m p ing facility that is som ew hat d ifficu lt to
w ork w ith, bu t exam p les m ay be fou nd below . snoop is native to
Solaris, and its p acket cap tu re form at is the basis of RFC 1761. To
cap tu re UDP d atagram s:
snoop -x 0 udp <target>
Use debug on Cisco rou ters to sniff ou t sp ecific p rotocol issu es.
Ethereal from w w w .ethereal.com is free and p orted to all p latform s,
and is very p op u lar. N m ap , w hich is available for Wind ow s as
N m ap Win is from w w w .insecu re.org. Tcp d u m p , w w w .tcp d u m p .org,
is p orted to Wind ow s as WinDu m p .
Using nettl
nettl is a native tracing u tility on UN IX system s. An exam p le
com m and to initialize a nettl trace is:
nettl -tn pduin pduout -e <entity> -f /tmp/net
w here <entity> is the nam e of the Ethernet entity (d river) being u sed .
Get a list of these entities w ith nettl -ss. Typ ically it is BTLAN . A list
of the d rivers actu ally u sed on the system can also be obtained u sing:
ioscan -kf
The nettl com m and p rod u ces trace files nam ed / tm p / net.TRC000
and / tm p / net.TRC001 (circu lar files). In the case w here a w rong entity
is selected , the trace file stays w ith a size of 128 bytes. Stop the tracing
w ith:
nettl -tf -e <entity>
The binary trace files can be form atted w ith :
172
Fault Analysis Tools
netfmt -l -N -f /tmp/net.TRC00[0,1]
To red u ce the ou tp u t, d efine filters (see netfm t m an p age) on a file and
u se it on the netfmt com m and line w ith -c <file>.
173
16. Third Party Tools
N N M gained p op u larity as an SN MP m anagem ent p latform in large p art
becau se of its APIs. Ten years ago, d evice-sp ecific vend ors w eren‟t interested
in d evelop ing stand ard SN MP m anager featu res, and H P w asn‟t interested
in d evelop ing d evice-sp ecific m anagem ent featu res. That form u la w orked
w ell for m any years, bu t it has changed m ore recently. More recently, d evice
m akers are less and less interested in p rovid ing m anagem ent tools for even
their ow n p rod u cts and H P has resp ond ed by ad d ed d evices-sp ecific
m anagem ent su p p ort to N N M. H P m aintains a catalog of certified N N M integrated solu tions at:
http:/ / openview.hp.com/ partner/ isv/ ind ex.jsp
Using N N M APIs via the N N M Develop er Kit p rovid es access to the N N M
SN MP event stream s, allow s cu stom ized top ologies to be au tom atically
constru cted , and p rovid es p rogram m atic access to certain N N M legacy
d atabase d ata via C++ and JAVA p rogram m ing interfaces. All these sam e
p rogram m atic interfaces are also available in Chip Su tton ‟s CS-OV PERL
Mod u le available at: w w w .cs-net.com . Below are selected third p arty
solu tions, som e of w hich are certified by H P, and som e of w hich are not. This
is a sm all list that rep resents those solu tions that are m ost often m entioned
by N N M u sers on the Op enView Foru m u ser grou p listserv.
Environmental monitoring and contact management
Tools for m onitoring room tem perature or for interfacing w ith other analog
d evices and then conveying that inform ation to N N M includ e:
w w w.im ci.net
w w w.netbotz.com
w w w.uptim ed evices.com
w w w.om nitronix.com
w w w.sensorsoft.com
w w w.akcp.com
Freew are utilities for UN IX
Assorted tools used often by other N N M ad m inistrators inclu d e:
Expect
Sw atch
Putty (SSH )
VN C, only better
expect.nist.gov
swatch.sourceforge.net
ww w.chiark.greenend .org.uk/ ~sgtatham / putty
ww w.tightvnc.org
174
Third Party Tools
Web Load tester
I2Trace
ww w.w pid alam ar.com/ projects/ 123load test
ww w.geocities.com/ m ilicsasa/ Tools/ l2trace
Emulating X w indow s on Window s machines
Reflections and Exceed are the m ost p op u lar com m ercial tools for
em u lating X w ind ow s u nd er Microsoft Wind ow s system s. Freew are
tools that accom p lish this inclu d e Xm ing and Cygw in/ X. Users rep ort
Xm ing is easier to install and that Cygw in w ith X11 p ackages is m ore
highly configu rable.
Event correlation tools
Tools that su p p lem ent or rep lace the fu nctionality of N N M‟s bu ilt-in
event correlation engine, Op enView ECS inclu d e:
ECS-based solution provid er
SEC – Sim ple Event Correlator
InCharge
N erveCenter
w w w.logec.com
kod u.neti.ee/ ~risto/ sec
w w w.sm arts.com
w w w.openservice.com
SN MP and netw ork management tools
Tools that su p p lem ent or rep lace the fu nctionality of N N M‟s bu ilt -in
SN MP m anager featu res inclu d e :
SolarWind s SN MP tools
OpenN MS
N agios
SN MP Toolkit
Big Brother
Wind ow s Extension agents
TrapBlaster
N et-SN MP agent
MIB Brow ser
Bad ger TrapServer
w w w.solarw ind s.net
w w w.opennm s.org
w w w.nagios.org
w w w.taave.com
w w w.bb4.org
w w w.snm p -inform ant.com
w w w.realops.com
w w w.net-snm p.org
w w w.innerd ive.com
w w w.bad gerac.com
Topology and netw ork modeling tools
Tools that aid in top ology m ap p ing and m od eling inclu d e:
Ed ge enPortal and nVision
Mim ic SN MP Sim ulator
Am erigo and other tools
Opnet
w w w.ed ge-technologies.com
w w w.gam bitcom m.com
w w w.taave.com
w w w.opnet.com
Reporting and graphing tools
Tools that su p p lem ent or rep lace the fu nctionality of N N M‟s bu ilt-in
d ata collections and rep orting featu res inclu d e :
RRDtool
N MIS
Ploticus
w w w.rrd tool.com
w w w.sins.com.au/ n m is
ploticus.sourceforge.net
175
Fognet’s Field Guide to OpenView NNM
Cricket
OverTim e and PingTim e
StatSeeker
Kard inia
Concord eH ealth
MRTG
176
cricket.sourceforge.net
w w w.netic.com .au
w w w.statseeker.com
w w w.kard inia.com
w w w.concord .com
people.ee.ethz.ch/ ~oetiker/ webtools/ m rtg
17. Cisco Devices
This section covers m anaging Cisco d evices w ith N N M.
Cisco MIBs
The p rim ary reason to load Cisco MIBs is that they contain em bed d ed
translations for SN MP trap s that ap p ear in the alarm brow ser. The
easiest w ay to integrate Cisco MIBs into N N M is to load the
ap p rop riate CiscoWorks-N N M integration p ackage. This also p rovid es
cu stom sym bol m ap p ings for som e Cisco d evices. If only the MIB files
are of interest, they can be extracted from the integration p ackage and
load ed . Look for a file nam ed :
AllCiscoMIBs.m y
If the integration p ackage is not available, m ost top level Cisco
MIBs ship
w ith N N M. They can be fou nd
in the
$OV_SN MP_MIBS/ Vend or/ Cisco d irectory. Load them in d escend ing
ord er as follow s:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
SN MPv2-SMI
SN MPv2-TC
SN MPv2-TM
SN MPv2-CON F
SN MP-FRAMEWORK-MIB
CISCO-SMI
CISCO-PRODUCTS-MIB.m y
CISCO-TC.m y
EN TITY-MIB
CISCO-EN TITY-ASSET-MIB
The m ost u p d ated Cisco MIB's can be fou nd at
ftp :/ / ftp .cisco.com / p u b/ m ibs/ v2/ v2.tar.gz and
ftp :/ / ftp .cisco.com / p u b/ m ibs/ v1/ v1.tar.gz
177
Fognet’s Field Guide to OpenView NNM
CiscoWorks and N N M
For years, CiscoWorks (CW2K) requ ired the p resence of a netw ork
m anagem ent p latform su ch as N N M. CiscoWorks then becam e a
stand alone p rod u ct w ith a sep arate integration p ackage for sharing
d ata w ith N N M.
Sim ilar to N N M, CW2K is a collection of integrated toolsets. It is
com p rised of Cisco RME (Resou rce Manager Essentials) and CWSI
Cam p u s (Cam p u s Manager), w hich com bines CiscoView , Traffic
Director (form erly N etScou t RMON ), and ATM Director. In 2007, a
nu m ber of other p oint p rod u cts have been ad d ed to the Cisco Works
su ite. Also a nu m ber of Cisco netw ork m anagem ent softw are p rod u cts
for u se ou tsid e the CiscoWorks su ite have been released .
Cisco‟s “Third Party Integration Kit” p rovid es the N N M integration
elem ents. This p ackage ad d s CiscoView and Cam p u s Manager to the
N N M m enu s, and is intend ed to be u sed for integrating N N M and
CW2K w hen both sit on separate p latform s. Both Cisco and H P p rovid e
CW2K integration p ackage d ow nload s for N N M. They are su p p osed
to be id entical bu t there are ind eed d ifferences. Field consu ltants
rep ort few er p roblem s w ith the Cisco-originated integration p ackage
and claim the site is kep t m ore u p to d ate.
The internal integration u tility that ship s w ith CW2K is intend ed to be
u sed for integrating N N M and CW2K w hen both p rod u cts sit on the
sam e p latform . Both p rod u cts contain u n -integratable em bed d ed
d atabases and CW2K is m em ory-intensive, so it is recom m end ed to
ru n CW2K on a sep arate p latform from N N M.
If both are to be installed on the sam e p latform , set u p sep arate d isk
p artitions for each p rod u ct. The p rod u cts form erly u sed incom p atible
versions of JAVA for u ser interface, bu t in 2006, a p atch w as released
so CW2K su p p orts JRE 1.4. Once CW2K and N N M are integrated , the
recom m end ation is to tu rn off CW2K Availability Polling and setu p
N N M to forw ard trap s to DFM. IE brow sers behave best w ith CW2K.
See Page 22 for d etails on hand ling m u ltip le version of JAVA clients
w hen ru nning CiscoWorks and N N M on the sam e client or server.
CiscoWorks, N N M, JPI and Internet Explorer
CiscoWorks 2000 requ ires Java Plu g-In 1.3.1, and w hen a JPI 1.4.1
ap p let is started from the sam e instance of Internet Exp lorer, you can
get Internet Exp lorer errors. If you start CiscoWorks 2000 from ovw,
178
Cisco Devices
the system w ill try to u se the sam e brow ser w ind ow that w as last
lau nched . If that brow ser w as ru nning H om e Base (and hence JPI
1.4.1), Internet Exp lorer w ill throw a ru ntim e error. If you first start
CiscoWorks 2000, then lau nch H om e Base, you w ill see a d ialog box,
"Attachm ent to a ru nning Virtu al Machine failed ", follow ed by an
Internet Exp lorer crash.
The w orkarou nd is to start CiscoWorks 2000 in a sep arate Internet
Exp lorer instance. Start a new Internet Exp lorer (from the start m enu ,
not u sing File: N ew ) and brow se to http :/ / <m achine>:1741. This w ill
start CiscoWorks 2000. Start another new Internet Exp lorer w ind ow ,
then start H om e Base from the ovw m enu . This w ill u se the new IE
w ind ow , and leave the CiscoWorks 2000 ap p let in the old w ind ow .
Alternatively, you can start a new IE w ind ow after starting H om e
Base, so that CiscoWorks 2000 w ill start in its ow n instance of Internet
Exp lorer.
N N M and handling Cisco VLAN s
By d efau lt, Cisco sw itches store the CAM table for VLAN 1 in
the d ot1d Tp Fd bTable, w hich is the table N N M‟s d iscovery u ses to
bu ild level 2 top ologies. If there is nothing in that table,
i.e. d ot1d BaseN u m Ports retu rns 0, then N N M m ap s it as a “flat”
top ology. This can be overcom e by d iscovering the d evice w ith the
com m u nity string ap p end ed w ith @vlan_no w h ere vlan_no is a VLAN
in u se on the sw itch.
This can cau se trou ble for d iscovery if qu erying an N N M u nsu p p orted
d evice. Be su re to u se the latest Cisco Agents for the N N M version
u sed . See Page 219 for the URL to check ET d evice agents.
Cisco bu ild s sep arate brid ge tables for each VLAN . To view MIB d ata
for VLAN s, issu e snm p get or snm p w alk requ ests w ith the VLAN
nu m ber ad d ed to the com m u nity string, follow ed by the @ character.
For exam p le, the com m u nity p u blic@200 gives in fo on VLAN 200‟s
MIB d ata. N ote that VLAN 1004 is p resent on every Cisco sw itch
(d efau lt VLAN for FDDI). To get a list of VLAN s try one or the other
of:
snmpwalk vtpVlanState.1.1 <node>
snmpwalk.1.3.6.1.4.1.9.9.46.1.3.1.1 <node>
To get m ore granu lar VLAN traffic rep orts from Catalyst 6000
sw itches, setu p N DE w ith a netflow collector and configu re brid ged
featu res su ch as brid ged flow stats. N DE d etails can be fou nd at:
179
Fognet’s Field Guide to OpenView NNM
w w w .cisco.com / univercd / cc/ td / d oc/ prod uct/
lan/ cat6000/ sw _7_6/ confg_gd / nd e.htm
Another ap p roach is m ining the vlan.d at d atabase file from the sw itch.
This file contains d ata p ertaining to IDs, nam es, p ort m em bership ,
tru nks, VTP, etc. This is only configu ration d ata, how ever. If ru nning a
version of IOS less than 12.1(18.4)E or 12.2(20.4)S there w as a bu g
w here if the VLAN table w as em p ty, the d evice w ou ld generate
au thentication trap s. ET w ill p oll for VLAN s.
N N M’s Cisco discovery configuration feature
N N M V7.51 introd u ced the Cisco Discovery Configu ration (CDC)
featu re to help control and im p rove the sp eed of ET Discovery. This
p ow erfu l scalability featu re allow s the d ata p u lled from these d evices
to be controlled w ith resp ect to the IP Ad d ress and / or the SN MP OID.
Discovery of the follow ing entities can be controlled w ith CDC:
•
•
•
•
•
Forw ard ing Database (FDB) Tables
Per-Mod ule (Per-Board ) Inform ation for Multiple Board s
Interface to Board Mapping
VLAN Mem bership Inform ation
Port Aggregation
The CDP Agent com es w ith an m si-file for Wind ow s installation.
Enable CDC by cop ying the CiscoSw itchSnm p .u ser.cfg file from
$OV_N EW_CON F/ H POvCiscoAgt/ conf/ nnm et/ agents/ to the
$OV_CON F/ nnm et/ agents d irectory and m od ify the new cop y.
Before enabling CDC, note that w hile CDC can im p rove p erform ance,
it m ay d o so at the exp ense of accu rate top ology d ata and CFA d ata
that is u sed by the APA. It m ay be beneficial to d isable the Forw ard ing
Database (FDB) tables on strategic d evices. Doing so w ill force reliance
on only on CDP for connectivity, bu t connectivity to non -CDP d evices
from Cisco d evices w ill then not be d iscovered . For exam p le, if you
have a Cisco sw itch that is connected to a Wind ow s server, you w ill
not d iscover the connection to the server even thou gh the server is
m onitored by N N M. If d isabling the FDB tables is not p ractical, see
the section below on setting ET to give CDP p riority over FDB.
H P recom m end s exp erim enting on a few test d evices before d isabling
FDB tables m ore w id ely. They su ggest m onitoring the “N u m ber of L2
links” stat in the “View Top ology Statu s” tab u nd er “Discovery Statu s”
180
Cisco Devices
after ap p lying changes that affect CDC. When configu ring CDC, it is
very im p ortant to read and u nd erstand the H P White Pap er on CDC:
$OV_DOC/ WhitePapers/ CiscoDiscoveryConfiguration.pd f
In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using
Extend ed Top ology u ser m anu al.
Setting ET to prefer CD P over FD B connection data
If ET is enabled , ET w ill create m u ltip le connections on an interface
w hen a connection is rep orted both in CDP and FDB for the sam e
interface. N N M V7.51 Interm ed iate Patch 18 introd u ced a m ethod to
force ET to p refer a CDP connection over an FDB connection in case
m u ltip le connections are being rep orted from the sam e interface on a
Cisco
d evice.
To
enable
this
behavior,
configu re
the
m _UseInferenceLogic flag w ith a valu e of 17 in the follow ing file:
$OV_CON F/ nnm et/ DiscoSchema.cfg file.
Useful Cisco IOS SN MP commands
To d isable Cisco TCP connection reset trap s generated w hen a telnet
session to a Cisco d evice is reset:
snmp-server disable traps tty
See “Configu ring Cisco IOS Trap s” below for m ore on this. To force a
Cisco d evice to TFTP its config file, u se snmpset on w riteN et
(.1.3.6.1.4.1.9.2.1.55); it is equ ivalent to being logged on the Cisco
d evice and issu ing a 'w rite netw ork'. For exam p le:
snmpset –c <WriteCommunityString> <RouterName>
.1.3.6.1.4.1.9.2.1.55.192.168.1.1 octetstringascii
<TargetFilename>
The target d irectory m u st exist and have p rop er file p erm issions. To
set the sou rce ad d ress of SN MP trap s to be the loop back IF:
snmp-server trap-source loopback 0
The com m and to send a test SN MP trap , for exam p le, the
chassisAlarm On trap (trap nu m ber 6, sp ecific nu m ber 5):
test snmp trap 6 5
181
Fognet’s Field Guide to OpenView NNM
H ere are the com m and s to set rou ter to check avgBu sy1 every 60
second s and send s trap s on rising/ falling threshold s of 70/ 60 p ercent:
rmon event 1 trap RM7net description "High CPU Util"
rmon event 2 trap RM7net description "Low CPU Util"
rmon alarm 1 lsystem.57.0 60 absolute rising-threshold 70
1 falling-threshold 60 2 owner <system>
To d isable link u p / link d ow n from non -IP interfaces on a sw itch:
conf t
int gi 1/1
no snmp trap link-status
Assign or change ifAlias for an interface. Generally, ifDescr and
ifN am e are not w ritable. N ote that Cisco CAT OS d oesn‟t su p p ort
ifAlias, and w rites instead to the p ort nam e in the p ort table u nd er the
Cisco Stack MIB.
conf t
int fa0/1
description <newname>
ifAlias MIB is: m ib-2.ifMib.ifMibObjects.ifXtable.ifXentry.ifAlias
Advanced Board status via the APA
The APA p oller p rovid es board entity statu s and generates sp ecial
events relating to board s if p rop erly configu red . See p age 79 for m ore
inform ation on this featu re.
Command to create cut SN MP view s
The exam p le w ou ld cu t the rou ting table from being view ed by a
server nam ed hp ov. This w ou ld keep N N M from consu m ing too m u ch
CPU on rou ters w ith very large rou te tables, like ed ge rou ters. This can
also be accom p lished by setting the “-R” lrf op tion on netmon (see the
section on lim iting d iscovery on p age 39). More on this in the p ap er
referred to below on the u nd ocu m ented CPU p riority sw itch.
snmp-server view hpov internet included
snmp-server view hpov ip.21 excluded
snmp-server community <comm-string> view internet RO
Hex target IP for Cisco ping MIB
Use the follow ing PERL cod e to covert an IP Ad d ress to hexad ecim al
notation for setting the CiscoPingAdress octetstringhex using the MIB
Browser or using snmpset commands:
182
Cisco Devices
#! /opt/OV/bin/Perl/bin/perl
# convert IP address to hex
@i = split ('\.',$ARGV[0]);
$hip .= sprintf("%02X", $_) foreach (@i);
print "$hip\n";
Undocumented IOS commands
The u nd ocu m ented snm p -server com m and is u sed to set SN MP CPU
p riority:
snmp-server priority {low | normal | high}
The “Q” colu m n of “ show process” show s cu rrent p riority
This d oes not affect SN MP trap p rocess. For m ore, see:
w w w.cisco.com / w arp/ public/ 477/ SN MP/ ipsnm phighcpu.shm l
Use the follow ing u nd ocu m ented event logging control com m and for
toggling syslog event m essages:
logging event {link-status | subif-link-status}
The “no” form of the logging event link-status <interface>
com m and is u sed to tu rn off send ing u p , d ow n and change m essages
for an interface to the syslog. This is very u sefu l on live system s since
these system s generate so m any of these m essages. This is a
com p anion com m and to the d ocu m ented com m and w hich p revents
send ing the associated snm p trap : no snmp trap link-status.
Weird trap OID s (.1.3.6.1.6.3.1.1.5.4.1.3.6.1.4.1.9)
This is an SN MPv2 origin event OID. To resolve, create an OID alias in
trap d .conf. For a fu ll exp lanation, see the section on origin event OIDs
on p age 107.
Unknow n trap 1.3.6.1.2.1.0.1&2 from Catalyst 2950
The Cisco 2950 Catalyst sw itch IOS has a bu g that cau ses the u nknow n
trap . Up grad e the IOS. H ere is the text of the Release N otes for IOS
12.1_(9)EA1:
“When enabling traps on a catalyst 2950, m essages are received at t he N MS
w ith erroneous object id entifiers (.1.3.6.1.2.1.0.2 and .1.3.6.1.2.1.0.1)”
183
Fognet’s Field Guide to OpenView NNM
Cisco linkD ow n trap configuration
Som etim es, the TRAP m acros for Cisco MIBs d on‟t p rop erly d efine the
Cisco OID_ALIAS p rop erly, so Cisco Link d ow n trap s show u p in the
alarm brow ser w ith a m essage like:
linkDown trap received from enterprise cisco.1.469 with 4
arguments
The solu tion is to ad d an enterp rise id entification u sing the event
configu ration GUI for the .1.3.6.1.4.1.9.1.469 enterp rise. Alternatively,
m anu ally ad d the enterp rise to the $OV_CON F/ C/ trap d .conf file in
OID_ALIAS section, for exam p le:
OID_ALIAS ciscoProductsC2621 .1.3.6.1.4.1.9.1.469
Ru n xnmevents -event after ap p lying d irect tr ap d .conf ed its.
Controlling Cisco link dow n trap variable bindings
With IOS version 12.1, Cisco ad d ed su p p ort for RFC 2233
conform ance. This ad d ed su p p ort gives end -u sers a choice for w hat
sort of d ata is being p assed w ith the linkDow n and LinkUp trap s. RFC
2233 d efines linkUp and linkDow n trap s in the Interfaces Grou p MIB
m od u le (IF-MIB.m y) as follow s:
linkDown NOTIFICATION-TYPE
OBJECTS { ifIndex, ifAdminStatus, ifOperStatus }
::= { snmpTraps 3 }
linkUp NOTIFICATION-TYPE
OBJECTS { ifIndex, ifAdminStatus, ifOperStatus }
::= { snmpTraps 4 }
Prior to this change, Cisco‟s link d ow n/ u p trap s conform ed w ith RFC
1573, as d efined in the Cisco Interface Cap ability MIB m od u le (CISCO IF-CAPABILITY.m y) as follow s:
VARIATION linkUp -- TRAP-TYPE
-- OBJECTS { ifIndex, ifDescr, ifType, locIfReason }
VARIATION linkDown -- TRAP-TYPE
-- OBJECTS { ifIndex, ifDescr, ifType, locIfReason }
The ad vantage of RFC 2233 conform ance is that SN MP is su p p orted
for su b-interfaces. The d isad vantage is that the ifDescr, locIfReason,
and ifTyp e are not inclu d ed . Those objects m ay be m ore d esirable d ata
to have in the event m essage (for notifications, etc.). N ote that if u sing
the old er behavior, locIfReason shou ld be an arbitrary valu e for su binterfaces. The Cisco IOS com m and s to control this behavior are below :
184
Cisco Devices
Set RFC 2233 conform ance:
Set the old er behavior:
snmp-server trap link ietf
no snmp-server trap link ietf
Interface index remapping
N N M‟s m ore recent versions have featu res to d etect interface ind ex
rem ap p ing that m ay occu r after d evice reboots, etc. These rely on u se
of ifAlias, w hich requ ires consistent interface nam ing p ractices. ET and
the APA also have facilities to hand le ind ex re-m ap p ings, bu t these
algorithm s are not p erfect. The follow ing Cisco IOS com m and locks
ou t ind ex rem ap p ings and at a m inim u m shou ld be ru n on d evices
that are being p olled by N N M‟s d ata collector to facilitate consistent
rep orting:
snmp-server ifindex persist
ifSpeed and polling issues w ith data collections
Cisco's locIf*BitsSec are gau ge variables,
exp onentially d ecaying average com p u ted as:
being
a
5
m inu te
(previous m inute rate) + (the previous 5 m inute rate)
─────────────────────────────────
2
Unlike variables like ifInOctets, w hich are p u re cou nter valu es, the
p olling interval has no effect on the valu es stored by snm p Collect.
ifSp eed is a MIB-2 variable, and m ay not alw ays contain accu rate d ata
for the interface for Cisco d evices. Unless the sp eed is m anu ally
configu red in the rou ter, it shou ld be the d efau lt, w hich is T1/ E1 for
any serial connection, 10M for eth, 100 for fa, etc. If there are
su binterfaces configu red , they inherit the sp eed of the u nd erlying
p hysical p ort. With Fram e Relay, if the rou ter is connected to a sw itch,
the rou ter takes clocking from the sw itch, ignoring any configu red
sp eed setting. An incorrect ifSp eed configu ration w on't affect
transm ission in this case, bu t it m ay cau se errors for the rou ting
p rotocol in u se.
From an N N M p ersp ective, m ost u tilization collections and
p erform ance grap hs m ay show false d ata since som e of them u se
ifSp eed to calcu late u tilization.
Interesting Cisco MIBS, OID s and MibExprs
Use CISCO-CON FIG-MAN -MIB.m y to track config changes
Use CISCO-ISDN -MIB.m y for basic rate ISDN on access rou ter
185
Fognet’s Field Guide to OpenView NNM
Use CISCO-VPDN -MGMT-MIB-V1SMI.m y for VPDN Per-u ser
Cisco MIB's for Layer 2, CDP, and VLAN :
cisco-stack.m ib:
portIfInd ex, vlanPortMod ule, vlanPortVlan
cisco-cd p.m ib:
cd pCacheAd d ressType, cd pCacheAd d ress
cd pCacheDevicePort
cisco-vtp.m ib:
vlanTrunkPortManagem entDomain
cisco-vlan-m embership.m ib:
vm Vlan
Cisco system stats are m ostly fou nd u nd er the lsystem MIB tree.
CPU bu sy p ercentage in the last 5 second p eriod . N ot e that these
m easu rem ents are based on the last 5 second p eriod in the sched u ler,
not the last 5 realtim e second s. This m akes them better ind icators of
p otential issu es that arise from CPU consu m p tion:
.1.3.6.1.4.1.9.2.1.56
--busyPer
.1.3.6.1.4.1.9.9.109.1.1.1.1.3
1 m inu te exp onentially-d ecayed m oving average of the CPU bu sy
p ercentage:
.1.3.6.1.4.1.9.2.1.57
--avgBusy1
.1.3.6.1.4.1.9.9.109.1.1.1.1.4
5 m inu te exp onentially-d ecayed m oving average of the CPU bu sy
p ercentage:
.1.3.6.1.4.1.9.2.1.58
--avgBusy5
.1.3.6.1.4.1.9.9.109.1.1.1.1.5
m em %util "Percent available Mem ory on a Cisco Device Com puted by:
((TotalDRAM – freeMem )/ TotalDRAM) * 100”\
.1.3.6.1.4.1.9.3.6.6.0 \
.1.3.6.1.4.1.9.9.48.1.1.1.5.2 \
+ .1.3.6.1.4.1.9.9.48.1.1.1.6.2 \
+ .1.3.6.1.4.1.9.2.1.8.0 \
.1.3.6.1.4.1.9.3.6.6.0 \
- .1.3.6.1.4.1.9.9.48.1.1.1.5.2 \
+ .1.3.6.1.4.1.9.9.48.1.1.1.6.2 + / 100 *
PIX Firew all failover statu s:
.1.3.6.1.4.1.9.9.147.1.2.1.1.1.3 cfw H ard w areStatusValue variable from CISCO FIREWALL-MIB. It could be active(9) or stand by(10), d epend ing on the
prim ary unit's status.
186
Cisco Devices
What is exponentially-decayed moving average?
Com m on in stock charting as w ell as p erform ance m onitoring, this
typ e of correlated statistical averaging is also know n as exp onentially
w eighted or exp onentially d am p ed m oving averag es (EWMA), and is
com m only d ep loyed in Cisco SN MP agents. This statistical m od el
allow s for good com p ensation for lost d ata p oints or w ild sp ikes
com bined w ith the need to keep very few p ast sam p les in m em ory. In
essence, this m od el gives the highest w eigh ts to the m ost recent d ata
p oints, p rovid ing a good ind ication of general volatility. A concise and
straightforw ard exp lanation of how Cisco im p lem ents it can be fou nd
at: w w w .cisco.com / w arp / p u blic/ 66/ 3.htm l
Cisco temperature probes
N ot all Cisco d evices have variables that p ertain to environm ental
m onitors are instru m ented the sam e. For Catalyst 6000 series sw itches,
load the follow ing MIBs: CISCO-EN TITY-SEN SOR-MIB, EN TITY-MIBV1SMI. The OID for entSensorValu e is: .1.3.6.1.4.1.9.9.91.1.1.1.1.4.
Each entSensorValu e instance in the above variable m ap s to an
entPhysicalInd ex (1.3.6.1.2.1.47.1.1.1.1.2) for w hich an entPhysicalDescr
(1.3.6.1.2.1.47.1.1.1.1.2) is need ed . Som e Cisco d evices u se the
EN VMON MIB, inclu d ing Catalyst 5000‟s, 7x00 rou ters 12000 rou ters,
etc.
ciscoEnvMonTemperatureStatusIndex:
1.3.6.1.4.1.9.9.13.1.3.1.1
ciscoEnvMonTemperatureStatusDescr:
1.3.6.1.4.1.9.9.13.1.3.1.2
ciscoEnvMonTemperatureStatsValue:
1.3.6.1.4.1.9.9.13.1.3.1.3
ciscoEnvMonTemperatureThreshold:
1.3.6.1.4.1.9.9.13.1.3.1.4
ciscoEnvMonTemperatureLastShutdown:
1.3.6.1.4.1.9.9.13.1.3.1.5
ciscoEnvMonTemperatureState:
1.3.6.1.4.1.9.9.13.1.3.1.6
N ote that 3600 rou ters only rep ort ciscoEnvMonTem p eratu reState.
D irectly integrating Cisco syslog messages
A lightw eight version of H P‟s Op enView Op erations (OVO) logfile
encap su lator agent w as integrated into UN IX-only AE versions of
187
Fognet’s Field Guide to OpenView NNM
N N M 7.0 and later. This facility p rovid es a robu st syslog encap su lation
tool for p assing syslog entries to N N M alarm brow ser.
This facility is intend ed only for “occasional” u se, for exam p le for
traversing firew alls for few critical d evices w here SN MP cannot p ass.
Becau se of the ad d ed system overhead , it is not recom m end e d for u se
as a rep lacem ent for SN MP trap s, w hich are m u ch m ore efficient and
scalable. See p age 142 for d etails on the N N M syslog facility.
Cisco-specific event correlation circuits in N N M
The Event Classifier circu it classifies Cisco events into m eaningfu l
categories grou p ed by d evice. The chassis failu re circu it m onitors
Cisco trap s for tem p eratu re, fan failu re and p ow er su p p ly fau lts ,
consolid ating a set of environm ental tra p s into new Op enView
Enterp rise events.
Cisco links
MIBs:
w w w .cisco.com / public/ sw -center/ netm gm t/ cm tk/ m ibs.shtm l
Mibs via FTP: ftp:/ / ftp.cisco.com / p ub/ m ibs/ oid /
Mib Repository and trap translator:
w w w .cisco.com / cgi-bin/ Supp ort/ Mibbrow ser/ unity.pl
Configuring Cisco IOS Traps:
w w w .cisco.com / en/ US/ tech/ tk648/ tk362/ technologies_tech_note
09186a0080094a05.shtm l
Mib supported by VPN 3000 concentrator:
ftp:/ / ftp.cisco.com / p ub/ m ibs/ supportlists/ vpn3000/ vpn3000supportlist.htm l
Configuring VPN 3000 concentrator to send events as SN MP traps:
w w w .cisco.com / univercd / cc/ td / d oc/ prod uct/ vpn/ vpn3000/ 3_6
/ config/ events.htm
Syslog H ow -to:
w w w .cisco.com / en/ US/ prod ucts/ sw / cscow ork/ ps2073/ p rod uct
s_tech_note09186a00800a7275.shtm l
OSPF Configuration Managem ent w ith SN MP:
w w w .cisco.com / en/ US/ tech/ tk869/ tk769/ technologies_w hite_pa
per09186a00801177ff.shtm l
RAN CID (Really Aw esom e N ew Cisco config Differ):
188
Cisco Devices
(freew are for autom ated login to Cisco d evices)
w w w .shrubbery.net/ rancid /
DFM White Paper:
w w w .cisco.com / w arp/ partner/ synchronicd / cc/ pd / w r2k/ d vftm
n/ prod lit/ d igd f_w p.htm
CiscoWorks links
CiscoWorks Docu m entation
w w w .cisco.com / univercd / cc/ td / d oc/ prod uct/ rtrm gm t/ cw 2000/
H P-based N N M Integration Ad ap ter:
w w w .openview .hp.com / d ow nload s/ try_nnm _0001.htm l
Cisco-based N N M Integration Ad ap ter (CCO login requ ired ):
w w w .cisco.com / kobayashi/ sw -center/ cw 2000/ cm c3rd .shtm l
and w w w .cisco.com / cgi-bin/ tablebuild .pl/ cw 2000-u tility
CiscoWorks in Large-Scale environm ents White Pap er:
cisco.com / w arp/ pu blic/ cc/ p d / w r2k/ prod lit/ ckspp_w p.htm
Extend ed SAA agent su p p ort for MPLS/ VPN s:
w w w .cisco.com / en/ US/ prod ucts/ sw / iossw rel/ ps1839/ prod ucts_
feature_guid e09186a0080087b0a.htm l#w p 1037443
189
18. ovw Map Operations
ovw is th e foregrou n d p rocess th at lau n ch e s th e N N M static m ap .
Un d er UN IX, it is som etim es called th e Motif GUI. It m ay be
r eferr ed to as th e legacy GUI in fu tu re ver sion s sin ce H P p lan s to
d ep recate th at old er in terface in fav or of H om ebase, or JAVA -based
d yn am ic view s.
IPMAP is th e p rogr am th at r u n s to bu ild th e ovw 5-tiered top ology
based on OSI layer 3. ovtopmd is th e backgr ou n d p rocess th at feed s
statu s to IPMAP. IPMA P is th e ap p lication th at is registered w ith
th e ovw APIs to p r od u ce th e fam iliar lev el 3 top ology -based v iew
th at is broken in to a h ierar ch y of five su bm ap layers: Root, In tern et,
N etw ork, Segm en t, an d N od e.
Con tain m en t realm s a d d v irtu al layers at th e In tern et an d N etw ork
levels to allow u ser s to logically cu stom ize th eir top ologies. IPMAP
reads data from the map database and synchronizes it with data in the
topology database via ovtopmd. netmon provides direct status to interfaces at
the node level and the APA provides direct status via the status bridge
ovet_bridge to ovtopmd.
Map operations – selecting, cutting and pasting objects
Selecting m u ltip le objects can be tricky for m ap op erations for cu tting,
cop ying and p asting. If care is not taken, inad vertent cop y/ p aste or
cu t/ p aste op erations can be irreversible and u nrecoverable .
Som etim es, in ord er to recover certain m ap cu stom izations, d atabase
w ip es and netw ork red iscovery m ay be necessary. The follow ing
keyboard op erations are su p p orted :
Multiple select:
Multiple select:
Multiple select:
Move selected :
Cut/ copy – paste:
Cut/ copy – paste:
190
Control key and click on objects
Rubber band select
Rubber band and control-click d eselect
Left click and d rag
Right click popup m enu
H otkeys ctrl-x/ c/ v – use w ith care
ovw Map Operations
Tips for using the find operation
Most find op erations key off of N N M object d atabase field valu es. To
p eru se the available field valu es for a p articu lar object, Right -click over
a nod e in the ovw GUI and select object p rop erties. Any of the object
p rop erties su ch as cap ability flags can be u sed in search op erations.
Find op erations can also be m ad e based on sym bol p rop erties, so all
nod es that m ap to a certain sym bol typ e can be selected .
A very hand y w ay to select a bu nch of objects that are in m u ltip le
su bm ap s (to cop y and p aste to a container, for exam p le), is to ap p ly a
find op eration w hich highlights to search resu lts then select Ed it >Select H ighlighted .
Exact m atch of the string entered in the find field is the d efau lt
behavior for find op erations. N ote the exact m atch rad io bu tton in the
find w ind ow .
Regu lar exp ressions (RegExp ) are su p p orted in find op erations:
.
Any printable character
*
^
$
\
[ ]
Zero or more of the preceding character
If first, following chars match beginning
If last, following chars match string at end
Escape special characters
Ranges of characters
Exam p les of find strings:
192\.168\.[1-4]\.[0-255]
^.*\.bc\.ca
All IPs in subnets 192.168.1
through 192.168.4
Any nod e in the domain: bc.ca
Setting unset capability fields
Cap ability field s are u sefu l for find op erations, for filtering op erations,
and for bu ild ing rep orts u sing ovtopodump or ovobjprint.
Cap abilities that are u nset are not d isp layed .
Change this behavior by m od ifying the $APP_DEFUALTS OVw (see
p age 9) setting and changing the valu e of the below keyw ord to TRUE:
show UnsetCapabilitiesField s TRUE
191
Fognet’s Field Guide to OpenView NNM
Map object visual cues
Figure 18-1.
A pplication Plane:
User Plane:
Highlighted:
Selected:
Executable:
Transparent:
The IPMAP ap p lication p laced the object
Object can‟t be p laced logically in IPMAP
The resu lts of a Find op eration
The objects can be p assed to actions
Dou ble click on the object lau nches an
action as op p osed to op ening an u nd erlying
su bm ap
Object has bitm ap , bu t no “box.” This is a
d evelop er featu re u sed by third p arties
Summary of ov w submap properties
The follow ing attribu tes can be set for a su bm ap :
- Show Connection Labels
- Subm ap persistence
- Background Graphics
- Layout (auto, bus, star, ring, row / colum n, PTP, etc.)
- Wind ow Geom etry (size and placem ent of subm ap)
- Overlay (on| off, explod e into sam e or new w ind ow )
192
ovw Map Operations
N ew objects do not show up in maps
When u sing loadhosts (see p age 45) or w hen new objects show u p in
the object d atabase, as seen via ovobjprint, they som etim es cannot be
fou nd in any m ap s. A com m on reason this occu rs cou ld be that the
netw ork or the segm ent that the nod e is a p art of is u nm anaged . Use
find on the d evice‟s su bnet to confirm that the netw ork or segm ent
exists and is m anaged .
Tw o devices combine into a single node
There are several reasons for this, and several things to check w hen
this hap p ens. First, confirm that the DN S nam es assigned to all
interfaces on the tw o nod es have no overlap s. And , if u sing DN S, also
check if there is anything in / etc/ hosts, since it m ay be looking there
after failing to find an entry in DN S. N ext, check that the d evices d o
not have any interfaces that have the sam e IP ad d ress assigned . If this
w ere the case, a d u p licate IP ad d ress alarm shou ld have been
generated .
Du p licate IP ad d resses are not su p p orted u nless they m eet the
requ irem ents d iscu ssed u nd er OAD on p age 246. If an IP ad d ress had
been first assigned to one nod e, then re-assigned to the other nod es,
there cou ld be shad ow d ata that N N M is p icking u p t hat cau ses it to
m erge the nod es. Im p rop er su bnet m asks on the interfaces cou ld cau se
the d evices and N N M trou ble in attem p ting to learn the new
assignm ent d ata.
In ord er to resolve this situ ation, d elete the nod es from the top ology
and op en any closed N N M m ap s. Ru n: xnmsnmpconf –clearC. Search
the N N M d atabase u sing all the nam es associated w ith both nod es.
Use m enu bar Find -Object by Selection N am e.
H ighlight and d elete any retu rned resu lts and let N N M red iscover the
nod es. See p age 45 for inform ation on u sing loadhosts. If objects
p ersist and continu e to tu rn u p in object find resu lts, ru n : ovtopofix
as d escribed in the the p roced u re on p age 19. Ru n: snmpwalk on the
interface tables of the d evices.
On Cisco d evices, w alk the Ip Ad EntAd d r table. In som e cases, d evices
w ill retain old IP ad d resses w here they shou ld n‟t and rep ort these long
after an interface has been re-IP‟d . Using the IOS global com m and
"clear interface nam e/ nu m ber" w ill in m ost circu m stances clear the IP
ad d ress. If it is a d eleted virtu al interface, reconfigu re the interface,
assign an u nu sed IP ad d ress, clear the interface, and then d elete the
193
Fognet’s Field Guide to OpenView NNM
interface again. If none of these m ethod s w ork, a rou ter reboot w ill in
m ost cases clear the stale entries from the SN MP tables.
Controlling trunked or meshed connections in ov w
Mu ltip le connections betw een objects on a m ap are rep resented as
m etaconnections, w hich are connections that exp lod e to vie w the
u nd erlying m u ltip le connections. If it is p referable to rep resent these as
a single connection via u ser-sp ecified sp ecific p orts, this can be
accom p lished u sing the $OV_CON F/ netm on.equ ivPorts configu ration
file.
N ote that p rior to V7.51 Interm ed iate Patch 18, netm on cou ld n‟t
su p p ort m u ltip le p ort equ ivalence record s for a single host, bu t that
p atch resolved that and m u ltip le entries can now be m ad e in the file.
Understanding symbol and object delete operations
The key d istinction to help u nd erstand d elete op erations is that from
the GUI p ersp ective, sym bols are d eleted , not objects. From the
d atabase p ersp ective, objects are only d eleted if all sym bols
rep resenting that object are rem oved from all m ap s.
Objects d eleted from one m ap w ill not be rem oved from the object
d atabase if there are sym bols rep resenting that object in another m ap .
The “synchronizing” m essage that ap p ears w hen a m ap is op ened is
the op eration that keep s the m ap d atabase and the top ology d atabase
in sync.
This op eration reconciles objects that are m arked for rem oval in the
object d atabase w ith sym bols rep resenting them in that m ap ‟s m ap
d atabase. If there are no m ore sym bols left corresp ond ing to the
u nd erlying object, it is rem oved from the object d atabase.
"Delete From This Su bm ap " only d eletes sym bols from that one
su bm ap . For exam p le, a gatew ay object m ay exist in m u ltip le
segm ent‟s su bm ap s. "Delete From All Su bm ap s" m arks for d eletion
every instance of that sym bol from all su bm ap s, and all sym bols that
are contained if the sym bol is a container object.
The right-click m enu d elete is equ ivalent of "Delete from all Su bm ap s."
Finally, the follow ing com m and externally d eletes an object:
ovtopofix –r <node name>
194
ovw Map Operations
Fatal IPMAP or ovw init failed errors
These errors typ ically ind icate at least one corru p t ed object in the Map ,
Top ology, or Object d atabase. The p roced u re im m ed iately below
typ ically resolves the issu e. If not, it m ay be necessary to contact H P
su p p ort. It m ay also be necessary to d elete and rebu ild the d atabases
then red iscover the netw ork (see p age 12).
Find ing the offend ing object m ay not be that easy, how ever. Usu ally,
the ou tp u t of rep eated invocations of ovtopofix and ovw –mapcount
(u sing the op tions su ggested below ) can help id entify the object that is
cau sing issu es.
Remove a stubborn object from all maps and databases
Typ ically this is requ ired w hen an object is corru p ted becau se its
configu ration has d rastically changed and N N M hold s stale
inform ation abou t the object. Before an object can be d eleted from the
top ology and object DBs, it m u st be d eleted from all m ap s. Objects are
not d eleted from m ap s u ntil they are op ened . If an object is d eleted
from one m ap , it is m arked for rem oval in the object DB and shou ld be
rem oved from m ap s that su bsequ ently op ened d u ring the m ap
synchronization p rocess. Follow these step s to m ake su re that objects
are rem oved (in highly-scaled environm ents, rem ove the –v op tions to
increase p erform ance):
1.
2.
3.
4.
5.
6.
7.
8.
9.
Delete object using “d elete from all subm aps” option in ovw
Open all m aps and let them synchronize; close all maps
Run: ovstop netmon; then run ovw –mapcount –cruDRv
Repeat #3 until no errors are reported
Run: ovtopofix –chs0v
Run: xnmsnmpconf –clearCache, then start netmon, ovw
Confirm object is rem oved from DBs using find from ovw or by
running: ovtopodump –RISC <object>
If object is still in DB, run: ovtopofix –r <object>
Red iscover d evice using: loadhosts (see page 45)
Remove “REMOVED ” objects from databases
Typ ically, op ening all m ap s and ru nning the above p roced u re take s
care of m ost situ ations. Still, objects m arked for rem oval m ay still
p ersist. In this case:
1.
Select "Locate->Objects->By Selection Nam e" and enter REMOVED in
the "Regular Expression" field then hit "Apply" or "Enter"
195
Fognet’s Field Guide to OpenView NNM
2.
3.
4.
This find s all REMOVED objects and "H ighlights" them on the m ap
Click "View ->H ighlights->Select H ighlighted "
N ext click: "Ed it->Delete->From All Subm aps"
If the REMOVED objects still p ersist in the d atabase, stop netmon and
ru n the follow ing com m and s:
ovw -mapcount -RuD
ovtopofix -a
xnmsnmpconf -clearCache
Manually add objects
The com m and loadhosts (see p age 45) exists to au tom atically force
the d iscovery of objects that netmon cannot d iscover by itself.
loadhosts is typ ically attem p ted w hen netmon fails to d iscover
d esired objects. Use this p roced u re w hen loadhosts itself fails.
A typ ical exam p le is a d evice (p erhap s in a DMZ) that is on the other
sid e of a firew all from the N N M server. Com m only, the firew all
p erm its SN MP traffic, bu t blocks incom ing or ou tgoing ICMP traffic or
both. A sim ilar scenario is often seen w hen u sing MPLS or IPVPN
tu nnels/ clou d s w here access to an ISP's d evices is restricted . It is easy
to ad d an object for w hich the ap p rop riate segm ent -level su bm ap
alread y exists. Sim p ly choose Ed it-Ad d Object. It is trickier w hen the
segm ent or the netw ork sym bol above it d on‟t exist.
Before attem p ting su ch op erations, search the top ology u sing “find ”
for the IP su bnet to m ake su re it not hid d en from view or p laced in a
hard to find container in the top ology. When ad d ing a netw ork object
into the internet m ap , be su re to choose the “IP N etw ork” netw ork
sym bol from the Ad d Object p alette; IPMAP w on‟t recognize any other
sym bol typ e at this top ology layer. Give it a label and note that the
“Ad d Object” d ialog has the “IP Map ” Object Attribu tes available, and
not grayed ou t. If it is grayed ou t, som ething is am iss.
It‟s im p ortant to select and set the “IP Map ” attribu tes. Minim ally, set
the netw ork N am e, Ad d ress, and Su bnet m ask. All su bsequ ent entries
into this object‟s su bm ap m u st m ap to these attribu tes in ord er to be
recognized by IPMAP. A “verify” shou ld retu rn: “OK.” Once the
netw ork object is ad d ed , it shou ld not show u p in the u ser p lane.
196
ovw Map Operations
In ad d ition, it is a good p ractice to u se / etc/ netw orks or you r DN S to
enter inform ation abou t the netw ork, and the u se the -m op tion to
loadhosts to set the correct netm ask.
N ow , op en the netw ork object and create a segm ent object , w hich also
m u st be configu red w ith the p rop er “IP Map ” attribu tes su ch as su bnet
m ask. Op en the segm ent object, then ad d a generic nod e d evice, w ith
the “IP Map ” attribu te m inim ally containing the IP ad d ress, and then
d em and p oll it.
Externally manage or unmanage objects
Version 6.31 introd u ced the –g and –G op tions to the ovotopofix
com m and . Earlier version on N N M p ast v ersion 6.0 p rovid e su p p ort
for the -g and -G op tions via p atches.
Version 6.41 introd u ced the follow ing Dynam ic View s URL‟s w hich
allow objects to be m anaged or u nm anaged rem otely:
http:/ / <N N M-Server>::7510/ topology/ manage
http:/ / <N N M-Server>::7510/ topology/ unmanage
all N N M versions, the DIM com m and : xnmtopoconf –
manage/unmanage can be u sed to externally m anage or u n m ange
For
d evices, bu t this only w orks w ith AE version of N N M.
Overriding IPMAP symbol changes
IPMAP has rather narrow logic abou t m ap p ing constru cts. When a
sym bol is created by IPMAP, it is consid ered "m anaged " by IPMAP,
and any su bsequ ent sym bol ed its are su bject to IPMAP's review for
correctness. When netmon issu es a configu ration p oll, any changes to
su ch an object m ay be reset by IPMAP.
For exam p le, if the sym bol is u p d ated w ith a p referred nam e, the nam e
m ay change back to the original nam e that IPMAP chose for it. Use the
follow ing environm ent variable setting to p reserve m anu al settings of
IPMAP objects:
IPMAP_NO_SYMBOL_CHANGES=TRUE
Or, set the “-u ” op tion to the IPMAP ap p lication call in the IPMAP
registration file. See p age 5 for m ore on the ARF p rocess. See p age 3 for
m ore on setting environm ent variables.
197
Fognet’s Field Guide to OpenView NNM
Map status propagation rules
Defau lt:
This is d efined by the num ber of sym bols that are norm al, com pared to the
num ber of sym bols that are not norm al. Ad m inistrative status values are
ignored (Unm anaged , Testing, Restricted , Disabled .) The calculation is:
Unknown:
N ormal:
Warning:
Minor:
Major:
Critical:
N o Norm al or abnormal sym bols
All sym bols norm al
One sym bol abnormal; m ultiple n orm al:
Multiple sym bols abnorm al; m ultiple norm al
One sym bol normal; all others abnorm al
All sym bols are abnormal
Prop agate Most Critical
This setting sim ply takes the m ost critical status and propagates it from the
interface to the nod e, from the nod e to the segm ent, from the segm ent to the
netw ork, and from the netw ork to the internet sym bol in the root subm ap.
Customizing map status propagation rules
The u ser can d efine the p ercentage of sym bols in a given state that
cau se that state to be u sed . If tw o states are satisfied then the m ost
severe state shou ld be u sed . H ere are the d efau lt valu es for this op tion :
>5%
>10%
>20%
>30%
>0
else
Critical
Major
Minor
Warning
N ormal
Unknown
For exam p le, if 25% of the sym bols are Minor and 15% are Major then
Major is the statu s that is u sed , since it is the m ore severe statu s.
Symbol status
ovw su p p orts three d ifferent w ays in w hich color ( statu s ) can be set on
a sym bol: "Com p ou nd Statu s", "Sym bol Statu s" and "Object Statu s".
These op tions allow ap p lications and / or IPMAP control over the color
of the sym bol. To d eterm ine or change a sym bol‟s statu s, sim p ly right
click over the sym bol and select "Sym bol Prop erties".
Com p ou nd Statu s
Com pound status is used w hen the parent sym bol w ants to reflect the com bined
status of the sym bols in the sym bol's subm ap. e.g. Networks, Segm ents etc.
Sym bol Statu s
Sym bol Status is used w hen an application w ishes to have control over the color of
sym bol that may d iffer d epend ing on w hich subm ap the sym bol is on .
198
ovw Map Operations
Object Statu s
ovw propagates the status the sam e w ay to all the sym bols in the m ap that represent
this object. The value of this status is stored in the object in the map.
Symbol status influences
Statu s is stored in the m ap d atabase as w ell as in the top ology
d atabase. Map synchronization reconciles d ifferences.
IPMAP u ses the follow ing ru les to assign statu s sou rce :
- Interface sym bols are set to object status source.
- Devices in segm ents and networks are set to sym bol status.
- Segm ents and networks are set to com pound status.
- Routers in internet subm ap are set to com pound status.
N ote that this ru le su ggests that rou ter sym bol on a netw ork su bm ap
u ses sym bol statu s, w hile the sym bol on the internet su bm ap u ses
com p ou nd statu s. This m eans the rou ter sym bol on the netw ork
su bm ap reflects only the interface statu s of the rou ter in that netw ork,
how ever the statu s reflects all the rou ter's interfaces on the internet
su bm ap . See the ―-s compoundNodeOn‖ sw itch (d iscu ssed below ) to
change this behavior so all sym bols reflect the m ore global statu s of the
d evices.
Fine tuning ov w symbol status
IPMAP has op tions that allow som e interesting levels of granu lar
control w ith resp ect to statu s that goes beyond setting sim p le statu s
p rop agation ru les. The ―-s‖ sw itch to IPMAP affects a m ap behavior,
bu t only at the initial creation tim e of the m ap , in the follow ing w ays:
Connections reflect the status of und erlying interface:
-s connStatusOn
Connections d o not reflect the status of und erlying interface:
-s connStatusOff
Status of nod e sym bols reflects the status of all com ponents. This m eans that
sym bols on m ultiple levels and subm aps representing the sam e object reflect
m ore consistent status:
-s compoundNodeOn
199
Fognet’s Field Guide to OpenView NNM
Transient, persistent and on-demand submaps
To im p rove scalability, H P ad d ed the ability t o sp ecify the levels of
the IPMAP top ology w hich are load ed into m em ory u p on start u p
of the ovw m ap in version 5.0. This is called the
„on-d em and su bm ap featu re‟.
A transient su bm ap is one that is not load ed into m em ory and a
p ersistent su bm ap is one that is. Transient su bm ap s take longer to load
from the GUI and their contained objects are not available to ovw API
actions su ch as the ability to change their statu s p rogram m atically.
The p ersistence filter w as im p lem ented to allow filter-based ru les to
d eterm ine w hat su bm ap s are load ed . Persistent filters are m ostly
need ed by a few third p arty p rod u cts that requ ire their objects being in
m em ory. In highly-scaled environm ents, they can also be u sed to keep
su bm ap s containing p articu lar d evices in m em ory for qu icker GUI
access.
By d efau lt, all levels are p ersistent for UN IX installations. For
Wind ow s installations, only the internet level and the contain m ent
realm s are p ersistent.
Persistent levels are set throu gh m ap op tions in the ovw GUI or via the
“-d level” sw itch op tion to the ip m ap callb ack in the
$OV_REGISTRATION / C/ ip m ap ARF registration file. See the IPMAP
m an/ ref p age for sw itch level valu es and p age 6 for m ore on u p d ating
ARFs.
Care is need ed , thou gh, in m aking higher IPMAP levels transient. For
exam p le, rou ter sym bols on the internet su bm ap w hich w ou ld
norm ally take "com p ou nd statu s" cou ld p otentially not p rop erly
u p d ate their statu s since there m ay not be a nod e su bm ap in m em ory
from w hich to calcu late its statu s.
N ote that only p ersistent su bm ap s are inclu d ed in m ap snap shots.
setStatus utility
This contribu ted u tility sets the "object statu s" valu e of a m ap sym bol.
For setStatus to w ork, the sym bol rep resenting the object m u st have
its statu s sou rce set to “object”. Also, the ovw m ap m u st be op en in
ord er to change the statu s of the object. This m eans that a setStatus
200
ovw Map Operations
issu ed on an object w hen the m ap hap p ens to be closed is not
su bsequ ently changed w hen the m ap is eventu ally op ened . The p ath to
the scrip t is:
$OV_CONTRIB/NNM/setStatus/setstatus.ovpl
In ord er to m anu ally set a sym bol to allow its statu s to be changed by
the scrip t, right click on the icon and choose Sym bol Prop erties.
Change Statu s Sou rce to “Object.”
Occasionally, an error m ay be generated w hen th e object sou rce
d oesn‟t p rop erly m ap to the selection nam e of the target of the
setStatus com m and :
Could Not set status color for XXXX - Object not on map
This error can be su p p ressed by setting the follow ing $APP_DEF
xnm events resou rce (see p age 9 for m ore info):
warnOnUnknownSource:false
If setStatus is u nw orkable and insu fficient for externally
m anip u lating an object‟s statu s, it is very easy to change the statu s
sou rce for an object or a set of objects u sing the N N M APIs. See p age
174 for m ore on the ovw API.
Lining up symbols in a perfect line or row
Perfectly lining u p sym bols u sing m anu al d rag op erations is very
d ifficu lt. Another m ethod is to exp ort the m ap , then ed it the exp ort
file, m od ify the X and / or Y coord inates in the file to m atch the line
w hich the sym bols fall on, then im p ort the file back into the m ap .
N ote that ed iting the exp ort files is not su p p orted and cou ld resu lt in
m ap d atabase corru p tion if syntax errors are introd u ced throu gh
ed iting. Back u p the m ap d b p rior to attem p ting to ed it exp ort files. See
p age 273 for the p rop er backu p p roced u re.
Forcing icons to scale dow n in size
Consid er that each su bm ap is a d efau lt p ixel w id th and height. That
d efau lt afford s the largest icon that is registered . Things tend to get
clu ttered w hen there are a lot of icons are on a su bm ap . So, to m ake the
icons scale d ow n in size, the d efau lt p ixel w id th and height for the
su bm ap need s to be m ad e larger w hile keep ing the w ind ow size the
sam e.
201
Fognet’s Field Guide to OpenView NNM
To d o this, tu rn au to layou t on. If there are any backgrou nd grap hics,
they have to be tem p orarily rem oved becau se OV locks the p ixel w id th
and height to the backgrou nd . N ow , d rag the icons to the extrem e left,
right, top , and bottom of the w ind ow and notice that au to -layou t
au tom atically shrinks the icons as the p ixel w id th and height exp and .
Keep rep eating this p rocess u ntil the icons are sm all enou gh to get the
nu m ber of nod es d esired onto the su bm ap . Once this is com p lete, au to
layou t can be tu rned back off, backgrou nd grap hics can be re-ap p lied .
At this p oint, save the geom etry for the su bm ap in the View m enu .
Forcing icons to scale up in size
In general, tu rning au to layou t off and m anu ally sp read ing ou t the
icons m ay cau se the m ap to red raw the icons at a larger p ixel size. See
also the section above on lining u p icons to sp ecific XY coord inates.
The follow ing $APP_DEF p aram eter can be u sed to set the m inim u m
icon size for m ap s w ith m any objects:
OVw *layoutMinSym bol Rad ius
This setting is u sefu l w hen it is im p ortant to visu ally d isp lay the color
changes on very bu sy m ap s. Details on u p d ating $APP_DEF
p aram eters can be fou nd on p age 9.
Finding and closing open ov w maps
To d eterm ine how m any and w hich ovw m ap s are op en, ru n:
ovstatus –v ovuispmd
To d eterm ine w hich u sers have m ap s op en (UN IX only), ru n the
follow ing com m and , w hich m ay show u sers of the d efau lt m ap :
fuser -u $OV_DB/openview/mapdb/default/current/map.lock
To stop all ovw sessions:
ovstop ovuispmd
IPMAP tracing
Assign the nam e of a logfile to the follow ing environm ent variable and
IPMAP tracing is logged to that file w hen ovw is started in that
environm ent:
202
ovw Map Operations
IPMAP_LOG_FILE
See p age 3 for d etails on setting environm ent variables.
N etw ork named “10” or “arpanet” on internet submap
N N M nam es netw ork sym bols by u sing the nam e fou nd in
/ etc/ netw orks or ..\ system 32\ d rivers\ etc\ netw orks. The d efau lt
version of that file fou nd in an u nm od ified state on som e UN IX
system s (p articu larly H P-UX) has only the “10” netw ork d efined w ith
"arp anet" as the historical nam e. See p age 111 for a d iscu ssion of this
issu e.
The solu tion m ay be as sim p le as changing the d efau lt entry in
/ etc/ netw orks, then d eleting and red iscovering the d evices in the
affected netw ork. If a nam e for the corresp ond ing su bnet nu m ber isn't
fou nd in / etc/ netw orks, then N N M w ill nam e the netw ork sym bol as
the su bnet nu m ber.
N ote that N N M rem oves trailing zeros from the nam e. Therefore,
10/ 8, 10.0/ 16, 10.0.0/ 24 and 10.0.0.0/ 30 m ight all look like "10" w hen
nam ed . It m ay be a good id ea to assign d ifferent nam es in the
netw orks file to these netw orks if they exist. See p age 27 for m ore
d etails on how netw ork objects are nam ed . Also, avoid having any / 8 / 22 netw orks in you r netw ork.
N U0 interfaces turn red in map
In som e instances, nu ll interfaces are u sed for rou te ad vertising
aggregation, and they m ay be im p ortant d ep end ing on how the
netw ork is setu p . In m ost cases, these interfaces can safely be
u nm anaged . If there are m any, globally filtering them ou t u sing the
follow ing p roced u re m ay be an op tion:
If using the APA poller, set up an ifType filter to unm anage them (see page
86); if using netmon, use ovautoifmgr (see page 42). In either case, the
affected nod es should be rediscovered , or use netm on.interfaceN oDiscover
(7.5+), w hich is used by both the netm on and APA pollers.
Symbols superimposed on each other in submap
This is a bu g in N N M 6.2 and it w as fixed w ith a p atch . The p roblem
w as seen w hen d oing cu t-and -p aste op erations into su bm ap s w ith
au to-layou t tu rned off. This only affected only read -only m ap s.
203
Fognet’s Field Guide to OpenView NNM
Managing VLAN s
A d iscu ssion on m anaging VLAN ‟s for Cisco d evices can be fou nd on
p age 179. In general, VLAN SN MP info is virtu alized in the Brid ge
MIB via the SN MP com m u nity string. For exam p le, to list MAC
ad d ress for a VLAN for vtp VlanState.1.3, w here 3 is the VLAN
nu m ber:
snmpwalk -c passwd@3 <switch> .1.3.6.1.2.1.17.4.3.1.2
ET‟s VLAN View p rovid es su p p ort for a lim ited nu m ber of sw itches
and p rotocols. Cisco VLAN s are d iscovered via the brid ge MIB, the
Cisco Stack and CDP MIBs. Other vend ors that p rovid e good su p p ort
for VLAN View d iscovery inclu d e Enterasys, Extrem e, and H P.
Som etim es, interconnected sw itch es w hich carry VLAN traffic that are
not on the m anagem ent VLAN can cau se the N N M top ology segm ent
to change, d ep end ing on w hich sw itch is p olled . In this case, the
netm on.equ ivPorts file can be u sefu l in hard cod ing to p ort
connections from the N N M p ersp ective. See p age 194 for m ore
inform ation on u sing netm on.equ ivPorts. Also, assu re you have the
m ost u p -to-d ate ET d evice agents for you r critical connector d evices.
See Page 219 for m ore on d evice su p p ort.
Managing VPN endpoints
H ow N N M m anages VPN ‟s d ep end s on how the VPN behaves. If the
VPN is alw ays stable and p resent, it ju st looks like an IP tu nnel. Even
if there is at least one d evice on the other sid e of the tu nnel that is
p ingable and SN MP su p p orted , it p robably w on‟t be d iscovered by
netmon, bu t in this case, the reachable d evices can be load ed into the
top ology via loadhosts (see p age 45).
Other d evices on the rem ote su bnet m ay be d iscovered natu rally after
d em and p olling the first p ingable, SN MP-su p p orted object, bu t they
also m ay not. ARP u sage (requ ired for N N M to au to-d iscover) is not
stand ard across VPN s. Again, u se loadhosts to force everything to be
d iscovered .
If the VPN is not stable, those links m ight be m anaged in an ind irect
m anner, p erhap s via VPN -related enterp rise MIBs u sing an SN MP
d ata collection that generated threshold alarm s based on a change in a
p articu lar MIB table entry. Or, p erhap s the u p stream d evice m ight
su p p ort VPN -sp ecific trap s. Another p ossibility is that the u p stream
d evice can be configu red to log im p ortant m essages w ith the syslog
facility (see p age 142).
204
ovw Map Operations
Managing ISD N interfaces
Polling d orm ant backu p ISDN links m ay cau se them to activate. The
best p ractice is to set u p an ACL in the rou ter to block certain typ es of
traffic that cou ld activate them , i.e. ICMP, SN MP, RIP, etc.
Most vend ors‟ SN MP agents, inclu d ing Cisco, rep ort inactive ISDN
interfaces as “d orm ant” in the ifOp erStatu s MIB2 table. Setting the “ d ” oid _to_typ e flag (see p age 7) also restricts statu s p olling (via
netmon) to these interfaces. Observe that this flag w ill not restrict
p olling of d ow nstream d evices on the LAN segm ent connected to the
nod e w ith the ISDN interface.
Sw itching routers and routing sw itches in ov w
Also called m u lti-layer sw itches, there is an increasing p resence of
chassis w ith su p p ort for board s that rou te and board s that sw itch.
N N M‟s ovw m ap s have a hard tim e m ap p ing sw itches that contain
rou ter m od u les or rou ters that contain sw itching m od u les. This is
becau se those d evices often d on‟t consolid ate SN MP fu nctions. For
IPMAP top ology in general, level 3 d ata com es from RFC 1213 MIB2
tables and level 2 d ata com es from RFC 1493 brid ge MIB tables.
Extend ed Top ology w orks d ifferently in this resp ect and m ore d etails
on that can be fou nd on Page 229.
Som e of these d evices su p p ort sep arate SN MP agents for the rou ter
m od u le and the sw itch m od u le and m ay not be able to be ad d ressed
p rop erly via SN MP u nless sep arate IP ad d resses are assigned to the
d iscrete m anagem ent agents.
In som e cases, the d evice m ay be rep resented as tw o sep arate objects in
the ovw m ap s, p articu larly if sep arate nam es have been assigned to the
m anagem ent ad d resses in DN S. This m ay be d esirable d ep end ing on
the netw ork ad m inistrator‟s p references.
If this is not d esirable, u se DN S A record s for the uniqu e ad d resses,
bu t also assign CN AME record s that are the sam e - then netmon can
resolve m u ltip le ad d resses as being aliased to the sam e nam e. The
objects m ight have to be d eleted and red iscovered after su ch a change.
ET is better at rep resenting these d evices. In N N M 7.5, there is su p p ort
in ET for board -level d iscovery and m ap p ing, and this greatly help s
w ith this sort of d evice. While ovw m ap s m ay still have d ifficu lty
p rop erly rep resenting these d evices, the ET rep resentations shou ld be
ju st fine w ith this new er cap ability.
205
Fognet’s Field Guide to OpenView NNM
One issu e, thou gh, is that APA p olling m ay be read ing d ata from one
d iscreet agent or the other and the agent being p olled is not the d esired
agent. netmon d iscovery p icks a p referred SN MP ad d ress and the APA
u ses this ad d ress. See p age 44 for the p roced u re w hich exp lains how to
change the p referred SN MP ad d ress.
Wireless
N N M started su p p orting w ireless netw ork d iscovery in V6.4 (ET 2.0).
Wireless access p oints are d iscovered by netmon via SN MP and N N M
assigns the d evices the “isWireless” ovwdb attribu te.
Tuning ov w db for large numbers of objects
If ovobjprint -S show s m ore th an 5000 objects, u se the -n LRF flag to
ad ju st the d atabase to the p rop er size for the nu m ber of objects being
m anaged . The nu m ber shou ld be set to 5-10% above the nu m ber of
objects rep orted . See p age 5 for the LRF p roced u re.
Limiting ov w menu bar access for some users
The basic p rocess to lim it m enu bar access is to cop y the registration
tree to another location, p are d ow n the registration files in the cop ied
location by searching for the registration file entries that are to be
rem oved , then set the OVw RegDir environm ent variable to p oint to
the m od ified registration tree d irectory in the u ser login shells for the
u sers w ho are invoking the m od ified trees. To set environm ent
variables, see the p roced u re on p age 3.
On Wind ow s, be carefu l to set %OVw RegDir% in all u sers' p rofiles or
logon scrip ts, esp ecially the d efau lt p rofile for new u sers.
Limiting usage of read-w rite maps
N orm ally, the first instance of m ap is op ened in read -w rite m od e;
su bsequ ent m ap s are op ened in read -only m od e. Only one read -w rite
instance (ovw session) of a m ap can be op en at any one tim e. The
$OV_CON F/ ovw .au th file w as introd u ced in N N M 6.0 and is the
p referred m ethod for control u ser access to m ap s. This d oes not
p rovid e som e of the granu larity of control offered by the below
m ethod s, how ever. Ad d the follow ing OVw ap p -d efau lts settin g to
change the d efau lt m ap op en state to read -only:
OVw*readOnly:
true
For instru ctions on how to m od ify ap p lication d efau lts, see
p age 9. With this setting, read -w rite m ap s can be op ened only by
206
ovw Map Operations
exp licitly u sing the –rw op tion to the ovw com m and . Another m ethod
for forcing u sers to be able to only op en read -only m ap s is to alias or
hard cod e the ovw com m and in the u sers shell accou nt (UN IX), to the
ovw –ro com m and . Sim ilar tricks can be p layed in Wind ow s
environm ents. N ote that u sers w ho invoke ovw w ith the –rw op tion
get read -only m ap s if the m ap p erm issions d on‟t p erm it access via
ovwchown or ovwchmod com m and s (UN IX only).
Limiting access to certain N N M applications
N N M V7.51 Interm ed iate Patch 18 introd u ced the ovp rocess.allow
configu ration file in $OV_CON F to allow som e p rocesses to have
lim ited u ser-based access by u ser id . The ap p lications this ap p lies to in
V7.53 inlcu d e:
xnmsnmpconf
ovalarmadm
xnmevents
xnmtrap
Manipulating map ow nership and permissions
ovwperms is a front end to the follow ing three u tilities:
ovwchown
ovwchgrp
ovwchmod
allow s maps to associated w ith UN IX users
allow s maps to associated w ith UN IX groups
sets m aps w ith UNIX-like perm issions
These com m and s are not su p p orted u nd er Wind ow s. Any attem p ts to
try to d u p licate ovwperms-like activities on Wind ow s system s, for
exam p le trying to change ACL‟s on the N N M DB files or d irectories,
cou ld resu lt in d atabase corru p tion. N ote the com m and ovwls, w hich
m ay be fam iliar to u sers of old er N N M versions, is d ep recated in later
versions of N N M.
ovw .auth, ovw db.auth, ovspmd.auth, and ovserver files
The ovw .au th, ovw d b.au th , and ovsp m d .au th files w ere introd u ced in
N N M 6.0. They control, resp ectively, u ser access to the ovw GUI, client
access to ovwdb for rem ote consoles, and u ser access to the ovstop ,
ovstart and ovstatu s com m and s.
With N N M 6.0, the $OV_DB/ op enview / ovw d b/ ovserver file w as
introd u ced as w ell. This file is sim p ly a text file w ith the N N M
hostnam e and need s to be checked w hen restoring d atabases from
backu p s from hot stand bys, w hen hostnam es change, and w hen
m igrating N N M to new hosts.
207
Fognet’s Field Guide to OpenView NNM
Window s-specific monitoring tools in N N M Menus
Wind ow s N T Menu op tions are not actu ally d efined in any
registration files. These m enu entries seem to be generated by a
p rogram called ovwNTtools.exe. The execu tion of this file, how ever,
is
itself
d efined
in
a
registration
file,
nam ely
$OV_REGISTRATION / N Ttools. Move this file ou t of the registration
tree to d isable these Wind ow s-sp ecific m enu bar op tions. See p age 5
for d etails on the LRF Proced u re
Add right-click pop-up menu items to ov w GUI
The follow ing ARF ad d s a right-click p op -u p m enu item that issu es a
ping to the target nod e. Also, see notes on natping on p age 169.
Application "Popup_Ping"
{
Description { "HP OpenView", }
DisplayString "Ping";
/* Use -Shared to make sure only one instance of the process is
executed at any one time.
*/
Command -Shared "xnmappmon";
PopupItem <100> "Ping"
Context (AllContexts || isIP ||
isRouter || WantPopupMenus)
TargetSymbolType ANY f.action "natping";
Action "natping" {
MinSelected 0 ;
SelectionRule (isNode || isInterface) && isIP;
#ifdef NT
CallbackArgs "-helpBrowser nnm:netPingTsk\
-commandTitle \"Ping\" \
-appendSelectList –
`
appendSelectListToTitle \
-cmd Perl/bin/perl -S natping ";
#else
CallbackArgs "-helpBrowser nnm:netPingTsk\
-commandTitle \"Ping\" \
-appendSelectList –
appendSelectListToTitle \
-cmd natping ";
#endif }
Ad d the above into a file w ith any nam e u nd er the
$OV_REGISTRATION / C d irectory and check the syntax w ith the
com m and : regverify –arf <filename>. A restart of ovw sessions
is then requ ired .
Add menu item to launch SSH on selection (UN IX)
See above for fu ll p roced u re and u se this ARF snip p et in the action
block of the registration file:
Action "ssh" {
MinSelected 1;
208
ovw Map Operations
MaxSelected 1;
Command "/usr/dt/bin/dtterm -title "${OVwSelection1}" -e
ssh ${OVwSelection1}&"; }
Another ap p roach is to call a scrip t and p ass it a sp ecific u sernam e that
d iffers from the d efau lt, w hich in the above ARF w ill be the u ser w ho
op ened the ovw session. Also, su ch a scrip t cou ld not requ ire a
selection nam e so a host can be p rovid ed if a nod e is not selected . In
this case, sim p ly rem ove the “MinSelected ” requ irem ent from the ARF.
Su ch a scrip t m ight look like this:
#!/bin/sh
stty erase ‗^H‘
target=$1
if [ ―$target‖ = ―‖ ] then /usr/bin/echo ―Enter host to
connect to: \c‖
read target
echo ―‖
fi
/usr/bin/echo ―Enter Username: \c‖
read logon
/usr/bin/ssh $logon@$target
/usr/bin/echo ―‖
209
19. Web Interface
The Web GUI is based on legacy ovw GUI top ology and only p rovid es
visibility to legacy IPMAP top ology. This interface is to be d istingu ished
from the Dynam ic View s JAVA interface d iscu ssed in Section 21. The
legacy w eb interface requ ires an op en ovw session to be ru nning and the
u ser sessions are alw ays read -only view s. In the p ast, the Web GUI w as
a CGI-based ap p lication. With N N M 6.4, Jakarta Tom cat becom es the
op erative servlet.
URLs
N ote that the UN IX Web GUI access p ort changes from 8880 to 3443 in
N N M Version 6.4
Web GUI:
http:/ / nod enam e/ OvCgi/ ovlaunch.exe (Window s)
http:/ / nod enam e:3443/ OvCgi/ ovlaunch.exe (UNIX)
Lau nch a sp ecific m ap nam e :
http:/ / nod enam e:port/ OvCgi/ jovw .exe?MapNam e=MyMap
Lau nch a sp ecific m ap su bm ap nam e :
http:/ / nod enam e:port/ OvCgi/ jovw .exe?MapNam e=MyMap&ObjectNam e
=MySubm ap
Alarm Brow ser:
http:/ / nod enam e:port/ OvCgi/ ovalarm .exe
Rep orter:
http:/ / nod enam e:port/ OvCgi/ nnm ReportPresenter.exe
N N M V7.51 Interm ed iate Patch 18 ad d ed the follow ing ad d itional
URL
op tions
w hich
are
d ocu m ented
in
the
N etw orkPresenterURLop tion.p d f
w hite
p ap er
in
the
$OV_DOC/ WhitePap ers d irectory: ServerName, objectName and
PreferredSubmapType, for exam p le:
210
Web Interface
http:/ / hostnam e[:port]/ OvCgi/ jovw .exe?[MapNam e= <mapname>&Object
N am e=<selectionname>&ServerNam e=<servername>(Preferred Subm apType
=<submaptype> | Subm apN am e=<submapname>)]
Maintaining open ov w map sessions for w eb clients
Web clients requ ire op en ovw sessions. (n ote that Dynam ic View s d o
not requ ire op en ovw sessions). There are creative w ays to m ake su re
ovw sessions are available to serve w eb clients w ithou t having to
d ed icate a w orkstation to m aintaining ovw sessions for this p u rp ose.
xvfb:
The UN IX X Virtu al Fram e Bu ffer can be u sed to su p p ort ovw
sessions. xvfb is free and su p p orted by the X consortiu m and can be
d ow nload ed from w w w .x.org. Once installed , set u p an rc file to
au tom atically lau nch a session on system startu p . Don‟t forget to set
u p a kill scrip t as w ell. Make su re to give the nu m bered startu p link
file a higher nu m ber than the one u sed by the Op enView d aem on
startu p rc file. In the scrip t, enter:
/usr/bin/X11/Xvfb :10 &
DISPLAY=:10 ovw &
Or, a slightly m ore verbose version :
/usr/X11R6/bin/Xvfb :10 -screen 0 1152x900x8 2>>
/var/adm/Xvfb.log&
VN C:
VN C (also know n as REALvnc) is only one of m any virtu al
w ind ow ing p rod u cts. VN C is free, m u lti-p latform , and w id ely u sed
in the u ser com m u nity. It can be fou nd at: w w w .vnc.com
Also consid er a m ore actively-d evelop ed and stable VN C d erivative,
tightVN C: w w w .tightvnc.com . To configu re an ovw session to start
w hen the system starts u p or w henever ovstart is issu ed , ad d an LRF
file for ovw (d etails on LRF p roced u re on p age 5):
1.
2.
3.
4.
Create a LRF file, named ovw.lrf w ith the follow ing lines:
ovw:ovw:
OVs_YES_START:ovuispmd:ro:OVs_NON_WELL_BEHAVED:15:
Register the new LRF via: ovad dobj ovw.lrf
N ote the –ro option makes the session read -only. Map options can also
be add ed to open a particular map nam e.
The session can be controlled from other w ind ow s via ovstop ovw
and ovstart ovw com mand s
211
Fognet’s Field Guide to OpenView NNM
There is a very hand y new er op tion to the ovw com m and that checks
the health of XServer p eriod ically:
ovw -pollXServer
This op tion is also available as an OVw $APP_DEFS resou rce so it can
be invoked au tom atically (see p age 9).
On any virtu al w ind ow s server, m ake su re that the color d ep th is set
for 24. If it is set low er, ovw m ay generate the follow ing error: “Cannot
allocate 128 colors, Using m onochrom e im ages.” More abou t this error
can be fou nd on p age 20. There are several tools that can be u sed to
m ake su re the virtu al console stays locked . The follow ing site, for
exam p le, has a tool for Wind ow s that locks ou t the keyboard and
m ou se:
w w w.e-m otional.com/ tscreenlock.htm
An op en sou rce UN IX client to op en Wind ow s Term inal Server
sessions can be fou nd at: w w w .rd esktop .org
Web GUI login passw ord
The d efau lt u sernam e/ p assw ord for w eb GUI login is ovu ser/ ovu ser.
The p assw ord can be changed by ru nning:
ovhtpasswd <user>
Web GUI access using jovw registration files
Jovw registration files can be u sed to control WEB GUI u ser access
u sing stand ard htgrou p file d efinitions. In the htgrou p file, ad d grou p s
w ith the u sers need ed . In the jovw registration file ad d sections like
this:
<110> "map1 IP Network"
Access group1
Icon "launcher/network.16.gif"
ActiveHelp { "><Access to the Network Presenter which
displays the objects based on map1's Network."}
f.action ―map1‖;
Adding Web GUI menu and toolbar items
To ad d a toolbar bu tton, m od ify the jovw files in both:
$OV_WWW_REG/ Launcher/ C/ Jovw
$OV_WWW_REG/ Jovw/ C/ Jovw
Exam p le of a toolbar item to ping the selected d evice:
212
Web Interface
ToolbarButton <100> @"C/toolbar/newwin.24.gif"
Context "AllContexts" f.action "ping";
Action "ping"
{
MinSelected 1;
URL
"/OvCgi/webappmon.exe?app=IP+Tables&act=ping&help=on&sel=OV
wSelections;
WebWindow "ping"
{
Type limited;
Width 600;
Height 300;
}
}
N ote the u se of webappmon, w hich is the Web GUI equ ivalent of
xnmappmon (see p age 167). The cod e exam p le above ad d s the bu tton,
bu t u ses the sam e GIF as another bu tton on the toolbar. Change the
GIF by ad d ing the p ath to a 24x24 bit p ictu re.
Menu bar item s for the Web GUI are sp read am ongst several files. For
exam p le, the "p ing" m enu item in jovw is located in:
$OV_WWW_REG/ jovw / C/ snm pview er
Jovw m enu registration files u se the sam e syntax as stand ard N N M
ARFs (see p age 6).
Icons OK in ov w but not OK in w eb GUI
Som e third p arty p rod u cts d evelop ed for N N M create cu stom bitm ap s
for ovw (w hich su p p orts several grap hic form ats), bu t not in gif form at
for the Web GUI. The gif form at files cau se qu estion m arks to ap p ear
in the w eb GUI. The follow ing u tility can be u sed to convert the
bitm ap s to gif form at:
$OV_CON TRIB/ NN M/ ImageMagik/ convertBitmaps.ovpl
N ote that on UN IX, $OV_CON TRIB/ N N M/ Im ageMagik shou ld be
ad d ed to the LD_LIBRARY_PATH environm ent variable and
convertBitm ap s.ovp l need s to be ru n as the root u ser.
The new sym bols shou ld be d eceted right aw ay for jovw view s. To
force Dynam ic View s to recom p u ted its sym bol m ap p ings after
converting bitm ap s, ru n the ovdvstylereset com m and fou nd in
$OV_BIN and then refresh any op en d ynam ic view s to ap p ly the new
sym bols.
213
Fognet’s Field Guide to OpenView NNM
javaGrapher contributed data collection tool
javaGrap her is a contribu ted ap p lication and is a tool for live qu eries of
the snmpCollect d ata. It d oes not, how ever, p ick u p u ser-configu red
collections and is d esigned for m ore ad -hoc-typ e qu eries of SN MP
objects. It also d oesn‟t seem to w ork w ith the base installation of N N M
7.5. It is located in:
$OV_CON TRIB/ NN M/ javaGrapher
Running JAVA apps in event action callbacks
Use cau tion w hen calling JAVA ap p lications as au tom atic actions
associated w ith N N M events becau se a sep arate JVM is started for
each instance for each event lau nched and this can consu m e a lot of
system overhead . That said , the typ ical p roblem one encou nters is that
the classp ath is not set in the calling scrip t, for exam p le, the follow ing
error is seen in ovactiond .log:
Exception in thread "main"
java.lang.NoClassDefFoundError:
To resolve this p roblem , set the classp ath in the calling scrip t u sing:
java –classpath <path_to_script>
Secure JAVA-based telnet or ssh
In som e environ m ents, GUI u sers d on‟t have the ability to locally
access the d evices m anaged by N N M (jovw relies on this to lau nch
local u tilities for telnet, etc). There are a few free JAVA -based telnet
clients that fit nicely in this situ ation; here are links to tw o of them :
http:/ / javassh.org/
http:/ / w w w.m ud .de/ se/ jta/
Setting the preferred w eb brow ser
To configu re the brow ser u sed to lau nch URLs from w ithin N N M
m enu s and ad d itional actions, see the ovw eb.conf m an p age and u se
the $OV_CON F/ ovw eb.conf configu ration file. An exam p le entry
(UN IX):
Browser: /opt/ns-communicator/netscape -install %s
Port: 3443
To reu se an op en N etscap e instance rather than op en a new one:
Browser: /opt/netscape/netscape -remote "openUrl
(%s,_blank)" || /opt/netscape/netscape %s
214
Web Interface
Web GUI access control
User control for Web GUI access This can be achieved by creating
grou p s in the $OV_WWW/ etc/ htgrou p file w ith p articu lar u sers
assigned to a grou p . Then, by assigning an access clau se for that grou p
in the action block of a sp ecific action in the ap p lication's lau ncher
registration file, the action can be restricted to the u sers of the grou p .
The ap p lication's registration file for ovlau ncher is fou nd in the
d irectory:
$OV_WWW/ registration/ launcher/ C/
H ere is an exam p le $OV_WWW/ etc/ htgrou p file:
NetworkAdmin: Mary
NetworkOper: Rachel
NTOper: +
UNIXAdmin: Charlie
UNIXOper: Jake
OVAdmin: Charlie Jake
Su: Molly Max Goldie
NOC: Molly
H ere is an exam p le jovw registration file snip p et to restrict the d isp lay
of m ap op tions w ithin ovlau ncher to certain gr ou p s.
/*
* jovw.exe takes MapName and ObjectName parameters.
* MapName specifies the map for the Network Presenter
* to open, and ObjectName specifies the selection name
* of an OVW object to display.
*/
Action "NNMNetworkPresenter" {
URL "/OvCgi/jovw.exe?MapName=default";
WebWindow "OvWwwNetworkPresenter" {
Type limited;
} }
Action "NNMNetworkPresenterIP" {
URL
"/OvCgi/jovw.exe?MapName=default&ObjectName=IP+Internet";
WebWindow "OvWwwNetworkPresenter" {
Type limited;
}
Access NetworkAdmin;
}
/* The following two entries to the Action block are provided as
examples of how to add personalized maps. You must also update
the List blocks within this file.
*/
Action "NetAdmn" {
URL
"/OvCgi/jovw.exe?MapName=NetAdmin&ObjectName=IP+Internet";
WebWindow "OvWwwNetworkPresenter" {
Type limited;
}
Access NetworkAdmin;
}
Action "NNMNOC" {
215
Fognet’s Field Guide to OpenView NNM
URL
"/OvCgi/jovw.exe?MapName=NOC&ObjectName=IP+Internet";
WebWindow "OvWwwNetworkPresenter" {
Type limited;
}
Access NOC;
}
Action "NNMSu" {
URL
"/OvCgi/jovw.exe?MapName=Su&ObjectName=IP+Internet";
WebWindow "OvWwwNetworkPresenter" {
Type limited;
}
Access Su;
}
Action "NNMNetworkPresenterHelp" {
URL
"/OvCgi/OvWebHelp.exe?Content=cntref&Context=cxtref&Scope=sc
pref&Topic=Network_Presenter";
WebWindow "OvHelp" {
Type intermediate;
}
}
}
Web GUI and ovw auth files
Which rem ote system s that have access to the Web GUI can be controlled
throu gh the $OV_CON F/ ovw .au th and $OV_CON F/ ovw d b.au th files. By
d efau lt, all system s have access to the Web GUI. Both files m u st be
u p d ated if u sing these files to set granu larity to sp ecific nod es. Alw ays
m ake su re there is an entry for the N N M server itself in these files.
216
20. Extended Topology
Extend ed Top ology (ET) is literally an entire p rod u ct w ithin the N N M
p rod u ct. In 2000, the ET sou rce cod e w as licensed by H P from Riversoft
Technologies (Riversoft w as acqu ired by Microm u se in 2002 and Microm u se
w as acqu ired by IBM in 2005). H P has been integrating ET into N N M in
stages ever since and ET has increasingly becom e the d om inant architectu ral
com p onent of N N M in term s of u ser interface d evelop m ent and
rep resentation of new top ologies. This has m anifested itself as Dynam ic
View s in N N M (See section 22). In essence, Extend ed Top ology p rovid es
visibility to Layer 2 connectivity, H SRP, VRRP, VLAN S, IPv6, and
overlap p ing ad d ress sp aces that are otherw ise im p ossible w ith legacy N N M.
When N N M is first installed , Extend ed Top ology is not enabled . The reason
for this is that in highly-scaled environm ents, zones m u st be configu red or
ET m ay becom e a p erform ance bu rd en. Also, a su ccessfu l ET d iscovery is
d ep end ent on a w ell-d iscovered netmon-based top ology, and it is a good id ea
to take a hard look at the resu lts of netmon d iscovery before enabling ET.
Several red iscoveries of netmon-based top ology shou ld be exp ected d u rin g
the im p lem entation p hase of m ost N N M d ep loym ents.
Enabling Extended Topology
Read the gu id e to Using Extend ed Top ology thorou ghly before
enabling ET. When ru nning for the first tim e, u se:
$OV_BIN/setupExtTopo.ovpl
If this com m and is being ru n for the first tim e, it p rom p t s for zone
configu ration. In environm ents w here there are few er than 200
connector d evices, selecting au tom atic zoning shou ld be OK. Zone
configu ration can alw ays be u p d ated throu gh the Extend ed Top ology
Configu ration w ind ow available from the Discovery Statu s tab in
H om ebase.
In general, zoning in sm aller N N M d ep loym ents w ill ad versely affect
p erform ance. The best p ractice is to avoid u sing m ore than one zone.
217
Fognet’s Field Guide to OpenView NNM
Dep end ing on the environm ent (and the version of N MM), d iscovery
m ay take som e tim e. It can be m onitored in H om ebase u nd er the
Discovery Statu s tab.
ET autozoning
Au tozoning is p rom p ted for by setu p ExtTop o.ovp l. The zones created
by this featu re are strictly based on Layer 3 inform ation from the
classic N N M top ology. All su bsequ ent top ology d iscovery ru ns w ill
not u se the au tozone featu re and w ill then p lace any new ly -d iscovered
d evices into the d efau lt zone.
A best p ractice, then, is to m anu ally ru n the au tozone scrip t before
ru nning a new ET d iscovery. By d efau lt, a new d iscovery is ru n w hen
a threshold of new ly d iscovered nod es is reached , bu t the d iscovery
can be ad ju sted in the H om eBase extend ed top ology configu ration
w ind ow . The au tozoning featu re can be lau nched u sing the
“Configu re zones au tom atically” op tion in the extend ed top ology
configu ration w ind ow or by lau nching the follow ing com m and :
$OV_BIN/ autozone.ovpl
Monitor p rogress in the $OV_PRIV_LOG/ au tozone.log file. Restart
ovas after ru nning au tozoning u sing ovstop/ovstart. Test the zones
u sing the test zones op tion in the extend ed top ology configu ration
w ind ow .
If ru nning N N M on m ore p ow erfu l hard w are, the nu m ber of nod es
that are p laced in a single zone can be increased by m od ifying the
numManagedObjects p aram eter in the follow ing file, bu t H P
recom m end s not u sing a valu e of m ore than 10,000:
$OV_CON F/ nnm et/ recVals.conf
Manual zoning tips
If exp eriening trou bles w ith the au tozone featu re, consid er creating the
zones m anu ally. Use these gu id elines for the zones:
Create one zone for core netw ork d evices
Create one or m ore zones for the central site‟s L2 connecting
d evices and key com p onents
218
Extended Topology
Create one or m ore zones for external sites L2 connecting d evices
and key com p onents
Use d evice nam ing stand ard s as w ild card s in the zone
d efinitions. Su p p orted w ild card s in zone d efinitions are:
*
?
[..]
[!..]
n-n
Any num ber of characters up to the next period (.)
A single character
A single character or a range of characters [acd ] or [a -f]
Strings NOT containing a character or range [!acd ] or [!a -f]
From -to for ranges of IP ad d resses
Rem em ber to test all zones u sin g the test zones op tion in the extend ed
top ology configu ration w ind ow . The ou tp u t files created by this
op eration is $OV_TMP/ etzonetest.ou t and this file contains a list of all
the d iscovered nod es and w hat zones they are p laced in.
Restart or force ET discovery
See p roced u re on p age 10
ET device agents
Extend ed Top ology m akes d evice-sp ecific qu eries to su p p orted
d evices to bu ild connectivity, top ology, and intelligence for the p olling
engine. The follow ing URL‟
N N M d evice and protocol support:
w ww .openview .hp.com/ prod ucts/ nnm et/ support/ d evice_support.htm l
N N M d evice support:
openview.hp.com/ prod ucts/ nnmet/ support/ d evice_requirem ents.htm l
ET discovery and SN MP configuration
SN MP com m u nity strings are tran sferred to N N M/ ET from the legacy
SN MP configu ration su bsystem , bu t in V7.53 and p rior, SN MP p olling
tim eou t and retry valu es are not transferred . In V7.53, tw o op tions are
available for cu stom izing tim eou ts and retries in ET: the first is to
im p ort the settings from legacy SN MP configu ration; the second it to
configu re ET Discovery p olling tim eou ts and retries ind ep end ently.
To enable im p orting of classic SN MP configu ration tim eou ts and
retries, ru n: ovstop ovet_dhsnmp then ad d the –useNNMSNMPConf
sw itch to the ovet_d hsnm p .lrf file p er the LRF p roced u re on p age 5,
then restart the ovet_dhsnmp p rocess.
219
Fognet’s Field Guide to OpenView NNM
For ET-based Discovery tim eou ts and retries, they are configu red in
the m _Tim eOu t and m _N u m Retries p aram ters near the end of th e
follow ing configu ration file:
$OV_CON F/ nnm et/ DiscoSnm pH elperSchem a.cfg.
For changes to take effect, restart the ET p rocesses u sing etrestart.ovp l.
The below logfile w ill ind icate if tim eou ts are occu rring d u ring ET
d iscovery:
$OV_PRIV_LOG/ ovet_d hsnm p.log
ET discovery considerations and limitations
ET m akes u se of d ata in the forw ard ing (FDB) tables of LAN Sw itches
w hen com p u ting top ological relationship s. Many sw itches d o not
generate layer-2 p ackets that contain their ow n layer -2 ad d resses. This
resu lts in the forw ard ing tables of neighboring sw itches not having
entries for each other. This resu lts in too m any sw itch-to-sw itch
connections to be created by ET w hen sw itches are not d irectly
connected to each other.
ET can take ad vantage of p rotocols like CDP to elim inate non-neighbor
connections in the top ology. Su ch p rotocols "ad vertise" the p hysical
ad d resses via layer-2 p ackets. This resu lts in u sefu l connectivity
inform ation being available in SN MP forw ard ing tables. Also, w hen
VLAN s are in u se in an environm ent (and d iscoverable via su p p o rted
vend or-sp ecific SN MP agents), "non-neighbor connections" can often
be recognized as su ch. Below are ad d itional consid erations regard ing
sw itches:




220
Exp ect H P to d evelop im p rovem ents to the "top ology stitching"
algorithm s in fu tu re releases to N N M, keep u p -to-d ate on N N M
p atches and on ET d evice agents.
With Cisco d evices, CDC shou ld be u nd erstood and cu stom ized
for the environm ent. See p age 180 for m ore inform ation on CDC.
ATM sw itches that su p p ort the ILMI MIB p rod u ce m ore
accu rate and com p lete ATM sw itch connectivity . The ILMI
d iscovery agent only d iscovers rem ote neighbors for interfaces
w ith ifTyp e of atm (37). Devices su p p orting the ATM MIB that
rep ort ILMI neighbors for other interface typ es, for exam p le
sonet(39) w ill not be d iscovered .
Extrem e N etw orks d evices ru nning OS version 6.1.7b7 or later,
enable SN MP access to the forw ard ing d atabase:
Extended Topology
enable snmp dot1dTpFdbTable
ET d iscovery m ay treat certain rou ters as "end -nod es" u nd er
circu m stances w here there is not enou gh d ata to create rou ter-to-rou ter
connections. ET in N N M V7.53 and p rior d oes not have su p p ort for
the Rep eater MIB (rfc2018) or for enterp rise-sp ecific rep eater and / or
hu b MIBs. N N M, how ever, continu es to interrogate d evices
su p p orting rfc2018, as w ell as the H P Interconnect Fu nction ality (ICF)
MIBs, w hich su p p ly connectivity inform ation for N eighbor and N od e
View s.
Improving ET discovery accuracy
The follow ing step s m ay im p rove the accu racy of the top ology
d iscovered by ET:
1.
2.
3.
4.
Enable d iscovery protocol (CDP, FDP, EDP) to im prove accuracy
If the above protocols aren‟t available, run d iscovery d uring an active
tim e on the netw ork to better grab Forward ing Database tables in
sw itches
Consid er the need to m anage end -nod es. Managing end -nod es
im proves L2 connectivity accuracy but may cause more events
Check DN S perform ance. ET uses DN S lookups extensively
The CDC introd u ced in V7.51 (See Page 180) m ay affect the accu racy of
d iscovered d ata. The CDC is not enabled by d efau lt how ever.
Improving ET discovery performance
The p erform ance of ET d iscovery and the APA p oller has been
d ram atically im p roved throu gh several version s of N N M and
p articu larly throu gh increm ental p atches to V7.5. When encou ntering
issu es w ith p erform ance alw ays m ake su re the latest version and the
latest p atches are installed . As w ith netmon, ET d iscovery p erform ance
is also d irectly affected by DN S p erform ance. See Section 4 on DN S for
m ore on this.
N N M V7.51 introd u ced CDC to p rovid e ad m inistrators a w ay to
control how m u ch d ata is p u lled from Cisco d evices d u ring ET
d iscovery. This featu re can greatly im p rove the p erform ance of ET
d iscovery, bu t d oes so at the p otential exp ense of accu rate top ology
rep resentations. See Page 180 for d etails on the CDC.
Interface filtering, w hich w as introd u ced in a p atch to V7.5 can greatly
sp eed ET d iscovery in highly-sw itched environm ents. More
221
Fognet’s Field Guide to OpenView NNM
inform ation on this featu re can be fou nd in the InterfaceFiltering w hite
p ap er and on the section on the netm on.interfaceN oDiscover file on
Page 53.
Dynam ic View s by d efau lt u ses a com bination of filters from ET filter
files (p aConfig.xm l and Top oFilters.xm l) and the legacy N N M "filters"
file ($OV_CON F/ C/ filters). Disabling the legacy filters file sim p lifies
filters d efinitions and boosts p erform ance. If ru nning N N M versions
p reviou s to N N M 7.53, change the "enableETFilters" p aram eter from 0
to 1 in:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l
If ru nning N N M 7.53, change the "enableETFilters" p aram eter from 0
to 1 in:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ 1Nod eServlet.xm l
Reducing ovet_poll startup time
If startu p tim e for ovet_p oll is taking m ore than a m inu te or tw o, there
m ay be a large nu m ber of u nconnected interfaces in the environm ent.
The follow ing setting m ay red u ce startu p tim e:
Ed it $OV_CON F/ nnm et/ top ology/ filter/ Top oFilters.xm l
Find the section below and com m ent it ou t:
<!-<filter name="NotConnectedIF" objectType="Interface"
title="NotConnectedInterfaces" description="Interfaces
without Layer 2 Connection">
<operator oper="NOT">
<filterName>ConnectedIF</filterName>
</operator>
</filter>
-->
It is com m ented by ad d ing <!-- as com m ent start, and --> at the end .
Then, locate the follow ing line:
<interfaceAssertion name="ConnectedIF"
title="ConnectedInterfaces" description="Interface with L2
Connection">
222
Extended Topology
And ad d this section above it:
<interfaceAssertion name="ConnectedIF"
title="NotConnectedInterfaces" description="Interface
without L2 Connection">
<operator oper="NOT">
<attribute>
<capability>isL2Connected</capability>
</attribute>
</operator>
</interfaceAssertion>
Check filter validity with ovet_topodump.ovpl –lfilt, the restart ovet_poll.
ET and ov w database synchronization issues
Trou ble can resu lt w hen objects are d iscovered by netmon, bu t have not
yet been d iscovered by ET. Prior to V7.51, there is no single com m and
to force a single object or set of objects to be d iscovered by ET. To force
a com p lete ET d iscovery, or a single zone d iscovery, see the p roced u re
on p age 10. To synchronize a nod e that has not been d iscovered by ET
bu t has been d iscovered by netmon in V7.51, see the p roced u re
im m ed iately below .
ET single/incremental node discovery
After V7.51, single nod es that are su bsequ ently ad d ed by netm on can
be m ad e a p art of Extend ed Top ology, bu t there are som e lim its in the
initial su p p ort for this featu re. For exam p le, connectivity for new ly
ad d ed nod es is not yet su p p orted . It is im p ortant to read the
Increm entalN od eDiscovery.p d f w hite p ap er for m ore inform ation on
u sing this featu re. In V7.53, this d ocu m entation is rolled u p into the
Gu id e to Using Extend ed Top ology u ser m anu al.
This featu re is d isabled by d efau lt. To enable this featu re, ad d the
follow ing LRF flag to the ovet_brid ge.lrf file and u sing the LRF
p roced u re on Page 5:
-single_nod e_d iscovery
Restart the ovet_bridge p rocess after u p d ating the LRF u sing
ovstop/ovstart. Prior to V7.53 Interm ed iate Patch 20, Single N od e
Discovery d iscovers interfaces on non SN MP nod es and m arks them as
d isabled .
223
Fognet’s Field Guide to OpenView NNM
Since the interfaces are u nconnected , p olling is controlled by a
p aConfig.xm l filter that u ses the new isN ew N od e and isN ew Interface
attribu tes on the new ly ad d ed nod es. These attribu tes are cleared after
a fu ll or increm ental zone d iscovery occu rs.
To initiate a nod e d iscovery after enabling this featu re, sim p ly issu e a
stand ard d em and p oll to the nod e u sing ovw m ap fu nction or from the
com m and line u sing nmdemandpoll. Increm ental nod e d iscovery
activities are logged to:
$OV_PRIV_LOG/ ovet_brid ge.log
D etect interface configuration changes
The below configu ration settings shou ld be consid ered if the APA is
enabled . Ad d ed in N N M V7.53, the follow ing LRF flag ad d ed to the
ovet_brid ge.lrf file (u sing the LRF p roced u re on Page 5) w ill enable
au tom atic d iscovery of interface config u ration changes:
-interface_config_change
Sim ilarly, the follow ing LRF flag w ill au tom atically d etect SN MP
interface ind ex table renu m bering:
-hand le_interface_renum ber
N ote that au tom atic d iscovery of interface changes and interface
renu m bering settings are d isabled by d efau lt. Restart the ovet_bridge
p rocess after u p d ating the LRF u sing ovstop/ovstart.
Detection of the above changes occu rs throu gh the APA‟s m onitoring
of the sysUpTime and entLastChangeTime MIB variables. By d efau lt,
the form er is en abled by d efau lt, bu t the later isn‟t. To enable
entLastChangeTim e m onitoring, ed it the p aConfig.xm l file and set the
entityMibEnable p aram eter in the ConfigPollSettings to “tru e.”
H P recom m end s that if the above p aram eter is set to tru e, a new filter
is created to id entify d evices that su p p orts the entLastChangeTime
MIB variable and then setting u p the ConfigPollSettings so that the
entLastChangeTime p aram eter is set to “tru e” only for su p p orted
d evices. Also, lim itations ap p ly to the classes of interfaces that the
above settings ap p ly to. The d ocu m entation for this m ay be fou nd in
the Gu id e to Using Extend ed Top ology u ser m anu al.
224
Extended Topology
D etect interface table additions
N N M V7.53 introd u ced the ifNumberQueryEnable p aConfig.xm l file
p aram eter setting in the config grou p w hich w hen set to tru e (false by
d efau lt), w ill cau se the APA to p oll the ifN u m ber variable w ith every
APA statu s p oll. If ifN u m ber increm ents, the APA w ill generate the
new event OV_APA_IF_ADDED.
Zone discovery tips and performance tricks
Zones are an ET scalability featu re and an absolu te necessity for
environm ents that m anage m ore than a few hu nd red connector
d evices.
As a general ru le, zones shou ld corresp ond to geograp hic locations or
other logical grou p ings w here the nu m ber of links betw een the grou p s
is m inim ized . Do not sep arate d irectly-connected sw itches, sw itches
connected to rou ters, or sw itches in the sam e global VLAN . Zones
shou ld be connected by having at least one rou ter from each zone in
another zone.
setupExtTopo.ovpl au tom atically p artition s d evices into d iscovery
zones. It su ggests zones that m aintain sw itch and rou ter connection,
m inim ize overlap and d u p licate SN MP qu eries, and im p rove
p erform ance. Zone configu ration occu rs after ET d iscovery, and is
based on the assu m p tion that the d iscovery w as com p lete and
accu rate. Manu al m anip u lation of the the zone of a p articu lar d evice
can be accom p lished by ed iting the follow ing file then restarting ET:
$OV_DB/ nnm et/ hosts.nnm
N ote that “0” is the d efau lt zone and the file is overw ritten w henever
ET d iscovery takes p lace (ed iting this file is not su p p orted by H P).
The sizing of the zones is configu red w hen enabling ET w ith
setu p ExtTop o.ovp l. The valu es u sed are stored in:
$OV_CON F/ nnm et/ recVals.conf.
The valu es u sed are based on the available RAM, sw ap sp ace, and the
nu m ber and sp eed of CPU's in the system . The valu es can be cha nged
after ru nning setu p ExtTop o.ovp l, bu t before ru nning Au tozone. N ote
changed
valu es
w ill
be
overw ritten
w hen
re-ru nning
setu p ExtTop o.ovp l. The valu es w hich give m ost p erform ance
im p rovem ents are: nu m Managed N od es and nu m Managed Object. Be
225
Fognet’s Field Guide to OpenView NNM
carefu l increasing them too m u ch, how ever, since lack of accu racy in
the top ology can resu lt.
Passw ord to access ET Configuration page
setupExtTopo.ovpl p rom p ts for a u sernam e and p assw ord w hen
ru n. This info is stored in ad m inistrator block in the follow ing xm l file
u nd er the m ain N N M installation d irectory:
$OV_AS\webapps\topology\WEB-INF\dynamicViewsUsers.xml
Connection editor
Introd u ced in V7.01, the conn ection ed itior allow s d irect m anip u lation
of the ET top ology d ata. H P ad d ed som e d ocu m entation on the
Connection Ed itor featu re in V7.51 w hich can be fou nd in :
$OV_DOC/ WhitePapers/ ConnectionEd itor.pd f
In V7.53, this d ocu m entation is rolled u p in Chap ter 2 of the Gu id e to
Using Extend ed Top ology u ser m anu al.
The d ocu m entation d iscu sses a m ore generalized ap p roach to
connection ed its that can be u sed to sp an nod es in sep arate zones and
u ses the follow ing file:
$OV_DB/ nnm et/ generalConnsEdits
Prior to v7.51, H P p rovid ed m ore a bit m ore lim ited connection ed it
fu nctionality throu gh the follw ing file:
$OV_DB/ nnm et/ connectionEd its
Only end p oints that alread y exist in top ology d ataba se can be
connected throu gh both files. The files contain OQL qu eries that
p erform the d esired op erations, for exam p le:
To ad d a connection betw een nod es Bobby and Jerry, both u sing
ifIndex 91:
insert into disco.connectionEdits (m_Name, m_NbrName,
m_Command) values ('bobby.dead.net [ 0 [ 91 ] ]',
'jerry.dead.net[ 0 [ 91 ] ]', 1);
To rem ove an invalid conn ection betw een nod es Bobby and Jerry, both
u sing ifInd ex 91:
226
Extended Topology
insert into disco.connectionEdits (m_Name, m_NbrName,
m_Command) values ('bobby.dead.net[ 0 [ 91 ] ]',
'jerry.dead.net[ 0 [ 91 ] ]', 2);
The last d igit in the target statem ent is the com m and and is one of
the follow ing:
1
2
0
Add connection
Delete connection
Do nothing
To u se the connection ed itor, Create, then ed it:
$OV_DB/nnmet/connectionEdits
Once the above file is created , ru n:
ovet_topoconnedit.ovpl
Discovery shou ld not have to be re-initiated , bu t if the exp ected
u p d ates d o not occu r, force an ET d iscovery p er p roced u re on p age
10. To d eterm ine the p rop er ifInd ex to sp ecify, u se:
$OV_SUPPORT/NM/ovet_topoquery getNodeByName <target>
To d u m p all the level 2 connection in the N N M/ ET top ology:
ovet_topoconndump.ovpl
End nodes not in same zone as connector
To d iscover connections for nod es that end u p in d ifferent zones from
their corresp ond ing connector d evices, m od ify the DiscoSchem a.cfg
file p er the instru ctions in the section on Connectivity in Chap ter 2 of
the Gu id e to Using Extend ed Top ology u ser m anu al.
Add connections for unsupported devices
ET top ology m ay not be com p lete, or, in som e cases, incorrect w hen a
d evice that is not su p p orted by the d evice agents is in p la y. In general,
the connection ed itor can be u sed to d elete inaccu rate connections that
m ay be assu m ed by ET from the d evice and ad d new connections to
p rop erly show the correct connections. An excellent exam p le is
p rovid ed in Chap ter 2 the Gu id e to Using Extend ed Top ology u ser
m anu al.
227
Fognet’s Field Guide to OpenView NNM
Layer 3 edge connectivity
Ed ge connectity is assu m ed by ET w hen a su bnet is shared by only tw o
ad d ressed interfaces. This assu m p tion m ay lead to erroneou s layer 3
connections in the ET top ology. To stop ET from creating connections
based on this assu m p tion, u ncom m ent the line that read s:
#enableConnectivity:1 and change “1” to “0” so the line read s:
enableConnectivity:0 in the follow ing file:
$OV_CON F/ nnm et/ Ed geL3Conn.cfg
Ru n a new ET d iscovery cycle p er the p roced u re on p age 217.
Exclude nodes from ET discovery
The brid ge.noDiscover file can be u sed to exclu d e netmon-d iscovered
d evices from being p laced in the ET top ology. Also, w ith the release of
N N M 6.31, H P introd u ced the ability of filters to be u sed to p revent
d iscovery of w hole classes of nod es based on stand ard N N M filters.
For m ore inform ation on Discovery Filters, see Page 39.
Show ing device details for non-SN MP devices
Inform ation for nod es that ET can‟t otherw ise d iscover can be
im p orted into the ET d atabase for som e lim ited field s inclu d ing
SysN am e,
SysContact,
and
u ser
d efinable
field s.
The
ETN onSnm p N od eDataIm p ort.p d f w hite p ap er (or, if ru nning V7.53,
the Gu id e to Using Extend ed Top ology u ser m anu al.) d escribes this
featu re, w hich requ ires 10 exp licit field s and is configu red via the
follow ing configu ration file:
$OV_CONF/nnmet/nonsnmpnodes.nnm.
Use the follow ing com m and to check the syntax of the configu ration file
after m aking changes to it:
$OV_SUPPORT/NM/testnonSnmpFile.ovpl nonsnmpnodes.nnm
Cu stom Icons/ Sym bols can be assigned to non -SN MP d evices by
u sing the follow ing sp ecial sysObjectID:
1.3.6.1.4.1.11.2.3.16.n
”n” is the OID instance assigned in the nonsnm p nod es.nnm that
corresp ond s to a sp ecific entry in that file.
228
Extended Topology
Sw itching routers and routing sw itches in ET
ET issu es cu stom SN MP p olls to d evices that are then u sed to convey
top ology, connectivity and statu s inform ation. Since m any d evices in
this class p rovid e m u ltip le m anagem ent card s, it is im p ortant that they
are configu red to best reflect the actu al u se of the d evice and its
relationship to other d evices in the netw ork.
In essence, if the d evice is set u p to fu nction m ore like a sw itch than a
rou ter, the m anagem ent card for sw itch fu nctions shou ld be enabled ,
and vice versa if the d evice acts m ore in a layer 3 rhelm . If the d evice
tru ly p rovid es both fu nctions, it m ay be p ossible, u sing DN S, to
ad d ress both m anagem ent card s as sep arate entities w hich w ill show
u p in the top ology as sep arate d evices. See Page 88 for m ore on ET
interaction w ith d evices of this typ e from an APA p olling p ersp ective
and Page 205 for m ore on netmon d iscovery consid erations and legacy
ovw im p lications.
ET command summary, support tools, log files
The follow ing com m and s are u sed to create and m anip u late the
Extend ed Top ology:
setupExtTopo.ovpl
autozone.ovpl
etrestart.ovpl –verbose –disco
ovet_topoconnedit.ovpl
Enable Extended Topology
Run autozoning
Force ET discovery
ET connection editor
The follow ing log files can be u sed to trou bleshoot issu es w ith ET:
$OV_PRIV_LOG/ovet_bridge.log
$OV_LOG/ovas.log
$OV_PRIV_LOG/ovet_disco.log
Bridge logfile
Tomcat server logfile
ET Discovery logfile
The su p p ort com m and s listed below can all be fou nd in the follow ing
d irectory: $OV_SUPPORT/ N M
$OV_BIN/ovet_topodump.ovpl
There is good docum entation in the m an/ ref pages for this com mand w hich
d um ps ET topology data.
ovet_demandpoll.ovpl
Poll nodes, update status bridge,, enable/disable polling, and tracing.
Note the -d (d um p) param eter changed in patch 15 to NN M V7.51 and gives
d etailed info on w hich filters are in use for interfaces and nod es. More on
page 72.
checkPollCfg
229
Fognet’s Field Guide to OpenView NNM
This tool displays the APA polling settings for a node, its interfaces,
addresses and boards that are configured in paConfig.xml. Note that the
displayed values only reflect settings in the paConfig.xml file, not the
runtime ovet_poll process.
ovet_topoquery
Use this versatile tool to d um p detailed d ata from the ET database. Also,
this tool can be used to change the status of ET topology objects w ith the
setNodeStatus and setIfState flags. Run this tool with the
und ocum ented –internal flag to d um p an extensive list of flags for this
com mand.
ovet_fixTopology.ovpl
This
tool
helps
fix
anom alies
th at
are
attributable
to
SN MP tim eouts d uring d iscovery. The script can be run w ith the
–checkDanglingConn option to see if the fix option is need ed.
ovet_generateTopoDeltaReport.ovpl
This tool com pares the current d iscovery pass w ith the last one.
ovet_ toponame2id.ovpl
This tool converts a ovw selection nam e to the ET object ID
ovet_ toposet.ovpl
This tool sets or unsets polling for a object, see page 75
ovet_reloadTopoDBTbls.ovpl
This tool d rops and recreates all N N M Extend ed Topology tables w ithout
d ropping the NN M data warehouse tables.
ovet_topoobjcount.ovpl
This tool outputs totals for d ifferent kind s of objects in the ET d atabase if
called w ith the –all option.
ovtopodbsnapshot.ovpl
This tool m akes a snapshot of the ET topology data .
dumpDiscoStatus.ovpl
Displays ET d iscovery status. All configured agents m ust get to state 4.
Valid state values are:
0: Und efined
1: Not Running
2: StartUp
3: Running
4: Finished
5: Died
dumpAgentProgress.ovpl
Displays ET d iscovery status for an ET agent.
ETsNoSnmpNodes.ovpl
230
Extended Topology
This tool outputs the IP ad d resses of nod es that d id no t respond to
Extend ed Topology setup SN MP query and can help id entify unresponsive
SN MP nod es.
ovet_truncatetopotbls_all.ovpl
This script is used to truncate all Solid tables used by the topology service.
It can be used to determ ine if there are interfaces that are m isconfigured in
ET.
misconfigIfaces.ovpl
This script find s interfaces w ith speed and / or d uplex m ism atch .
zoneTime.ovpl
This tool gives info about the tim e used for d iscovery d uring the last ET
d iscovery..
ET and D N S issues
Errors sim ilar to the follow ing m ay ap p ear in $OV_LOG/ ovet_au th.log:
2002-05-15 17:07:48 rvd: unrecoverable IP configuration error:
gethostname() returns '<hostname>',
gethostbyname() for that returns IP address '15.2.113.5',
but that address does not match any listed interface.
2002-05-15 17:08:07 rvd: startup aborted: Initialization failed.
Fatal warning : A critical bus error occurred. in ../CRivRvNet.cc at
line 367
These errors incicate DN S configu ration issu es and ovet_au th m ay d u m p
core and exit. Check the forw ard and reverse DN S configu ration for the
affected nod e.
231
21. Dynamic Views
Dynam ic view s are available throu gh the H om ebase GUI, w hich
p rovid es Extend ed Top ology rep resentations. Dynam ic view s are not
available u nless Extend ed Top ology is enabled , w hich it is not w hen
N N M is initially installed .
It is im p ortant to have a w ell-d iscovered netw ork top ology before
enabling ET. Dynam ic View s are controlled by the ovas d aem on, w hich is
registered throu gh the LRF p rocess. See p age 5 for tip s on controlling
LRF d aem ons.
H om ebase is able to p rop erly d isp lay the fam iliar level 3 rep resentations
typ ical of the legacy ovw or Web GUI, neither of w hich is architectu rally
able to su p p ort ET view s. H om eb ase is com p letely ind ep end ent of ovw
and the legacy d atabases that su p p ort it.
ET still relies on netmon for netw ork d iscovery (excep t for OAD
d iscovery). From netmon d iscovery d ata, only basic object d ata is
conveyed to the extend ed top ology d atabase, w hich ET u ses to again p oll
the d evices for ET configu ration once ET is enabled . ET p robes d evices
sp ecific agents for in -d ep th top ology d ata and H P m aintains a sep arate
su p p ort m atrix for extend ed top ology d evice agents.
Dynam ic View s has both server (ovas) and client (w eb p age w ith Java
Plu g-in) com p onents. Dynam ic View s featu res inclu d e:





Ad d and Delete nod es
Expand all or connecting neighbors, path
Poster printing: large view s can be printed in chunks
Toggle port labels
Tom cat-based au thentication
D ynamic view s available
Dynam ic view s available w ith N N M 7.5x. N ote that som e of these
view s requ ire ad d itional licenses. H SRP, VRRP, OSPF, and IPv6 view s
232
Dynamic Views
requ ire p u rchase of the Ad vanced Rou ting SPI. They are enabled by
d efau lt w hen N N M Evalu ation cod e is installed , bu t exp ire along w ith
the eval key.
H om e Base
Container View
ET Discovery Status
Alarm View (APA only)(T)
N eighbor View
Internet View
Segm ent View
Problem Diagnosis View
CDP View
IPV6 N etwork View
IPV6 Interface Detail (T)
Overlapping Add ress Domain View
Interface View (T)
(T) indicates table view
N ode View
Access Path View
Path View
N ode/ If Status(APA only)(T)
Station View
N etwork View
VLAN View
OSPF View
H SRP View (T)
IPV6 N od e Detail (T)
IPV6 Prefix Groups (T)
Container View
VRRP View (T)
V7.51 ad d ed container view s (Page 235) and access p ath view (Page
168). Several other view s can be ad d ed if u sing one or m ore of the
ad d -on SPI's for N N M. V7.53 ad d ed the Alarm View , N od e Statu s
View and Interface Statu s View . All three of these new view s are only
available in the APA is enabled .
D ynamic View s URLs
Dynam ic View s URLS changed betw een N N M V6.31 and
N N M V6.41. This is w hen the Tom cat server rep laced the
ovp l/ ov-cgi-based d ynam ic view s. The form er and new er locations are
resp ectively:
http:/ / <NN M-Server>:3443/ OvCgi/ ovlaunch.exe
http:/ / <NN M-Server>:7510/ topology/
Sp ecific view s can be accessed by URL d irectly, for exam p le:
http:/
http:/
http:/
http:/
http:/
/
/
/
/
/
<NN M-Server>:3443/
<NN M-Server>:7510/
<NN M-Server>:7510/
<NN M-Server>:7510/
<NN M-Server>:7510/
OvCgi/ nod eView.ovpl
topology/ nod eView
topology/ segm entView
topology/ ospfView
topology/ vlanView
233
Fognet’s Field Guide to OpenView NNM
D ynamic View s configuration
The follow ing tw o files are the m aster configu ration entry p oints for
Dynam ic View s:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l
$OV_CON F/ d ynam icView s.conf
With the release of N N M 7.53, m any configu ration p aram ters w ere
sp lit off from w eb.xm l and p laced in context-sp ecific xm l files in:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/
To m erge xm l files in the servletRegistration d irect ory w ith w eb.xm l,
ru n the follow ing com m and :
$OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl
D ynamic View s concurrent view s/users
The nu m ber of concu rrent u sers that can access Dynam ic View s is not
lim ited to the actu al nu m ber of u sers, rather to the nu m ber of d ynam ic
view s op en at any given tim e. acceptCount is the p aram eter w hich
d efines this m axim u m . Change the d efau lt by m od ifying the
acceptCount valu e for the connector classnam e for p ort 7510 in the
$OV_AS/ server.xm l file and then ovstop/ovstart the ovas d aem on:
<Connector className="org.apache.catalina.connector.http.HttpConnector"
port="7510" minProcessors="5" maxProcessors="125"
enableLookups="true" redirectPort="8443"
acceptCount="10" debug="1" connectionTimeout="-1"/>
D ynamic View s access via Webstart
On Wind ow s clients, JAVA Webstart can be u sed to access the
Dynam ic View GUI w ithou t u sing a stand ard w eb -brow ser. This is
d ocu m ented in the follow ing u ser m anu al: N etw ork N od e Manager
v7.5 Dynam ic View s ap p lication integration w ith Java Web Start
Ap p lication Manager. Webstart can be installed from :
http:/ / nnm -server:7510/ topology/ w ebstart
Problem s m ay arise in accessing w ebstart d u e to the fact that H P
p op u lates the .jnlp file w ith the short nam e for the server. If an
“Unable to load resou rce” error m essage is d isp layed , find the three
lines w ith the N N M server nam e in the follow ing file and change them
to u se the FQDN :
234
Dynamic Views
$OV_AS/ w ebapps/ topology/ w ebstart/ d vclient.jnlp
D ynamic view s access control
The follow ing scrip t p rovid es the front-end for m anaging u sers w ho
have access to d ynam ic view s:
$OV_BIN/ d vUsersManager.ovpl
By d efau lt, d ynam ic view s are op en to all, excep t for:

Configuring Extend ed Topology

Managing/ unm anaging nod es

Ad d ing/ d eleting nod es
To ad d a u ser role for an op erator or an ad m inistrator via Tom cat
realm s if ru nning versions of N N M p reviou s to 7.53, u ncom m ent the
block sim ilar to below in $OV_AS/ w ebap p s/ top ology/ WEBIN F/ w eb.xm l:
<security-constraint>
<web-resource-collection>
<web-resource-name>DV Access Secure</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>operator</role-name>
<role-name>administrator</role-name>
</auth-constraint>
</security-constraint>
If ru nning N N M 7.53, u p d ate the sim ilar block in the follow ing file:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ userAuthentication.xm l
Then ru n the follow ing scrip t to ad d you r changes in the
u serAu thentication.xm l file to the w eb.xm l file:
$OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl
Then ad d the u sers and assign roles in:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ d ynam icView sUsers.xm l
<tomcat-users>
<user name=―ovuser1‖ password=―mypw‖
roles=―operator‖/>
<user name=―ovadmin‖ password=―adminpw‖
roles=―administrator‖/>
</tomcat-users>
235
Fognet’s Field Guide to OpenView NNM
After any of the above changes restart ovas via:
ovstop ovas; ovstart ovas
The above au theticalion m ethod is BASIC (cleartext p assw ord s are sent
over the netw ork). To u se MD5 p assw ord s instead , enter the resu lts of the
follow ing com m and as the p assw ord above, w here <p assw d > below is the
p assw ord to be encryp ted :
―$OV_JRE"/bin/java -classpath \
"$OV_AS"/server/lib/catalina.jar:"$OV_AS"/bin/bootstrap.jar \
org.apache.catalina.realm.RealmBase -a MD5 <paawd>
N ote on IPF, the p ath for the com m and is: $OV_JRE"/bin/IA64N/java
Make su re LD_LIBRARY_PATH is set to:
$OV_JRE/lib/IA64N:$OV_JRE/lib/IA64N/server
N ext, ed it the $OV_AS/ conf/ server.xm l file and ad d the d igest p aram eter
of the <Realm > elem ent of the top ology Context to d igest="MD5”, e.g:
<Context path="/ topology" docBase="topology" d ebug="0">
<Realm classN am e="org.apache.catalina.realm .Mem oryRealm "
pathnam e="w ebapps/ topology/ WEB-IN F/ d ynam icView sUsers.xm l"
d igest="MD5"
/>
</ Context>
D ynamic view s via SSL
The follow ing Manu al w as ship p ed w ith N N M V7.53 to exp lain how
to enable SSL for lau nching Dynam ic View s throu gh Java WebStart:
$OV_WWW/ htdocs/ C/ manuals/ Configuring_SSL_for_Dynam icView s.pd f
Container view s
V7.5.1 introd u ced this p ow erfu l view set to rep lace the trad itional role
of ovw view s in rep resenting cu stom ized view s of the netw ork w ith
backgrou nd grap hics, etc. A w hite p ap er d escribing this featu re can be
fou nd in:
$OV_DOC/ WhitePapers/ ContainerAd m inistration.pd f
In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using
Extend ed Top ology u ser m anu al. Unlike ovw-based containers, ET
containers are not lim ited to the strict hierarchical constraints of
236
Dynamic Views
IPMAP. In fact, containers can even be contru cted u sing circu lar
references. Containers can be d efined d ynam ically u sing filters, w hich
are evalu ated each tim e the container is op ened . APA statu s is
p rop agated throu gh containers. The only real restriction on containers
is that they m u st be hierarchical and each container nam e m u st be
u niqu e.
The Container View s m enu s p rovid e a set of container creation and
cu stom ization op tions, w hich are w ritten to an xm l file:
$OV_DB/ nnm et/ containers.xm l
This file shou ld not be m anip u lated d irectly to create containers. Use
the follow ing p roced u re to m anu ally u p d ate the containers.xm ll file:





Backup the existing containers.xml file
Create a working copy of containers.xm l and ed it as d esired
overw rite the runtim e copy in $OV_DB/ nnm et w ith the working copy
Run ovstop ovas ; ovstart ovas
Reload the view s using the follow ing url option:
http:/ / <hostnam e>:7510/ topology/ home?reloadContainers=true
The w hite p ap er m entioned above contains an exam p le .xm l file and
d escribes the variou s tags allow ed in this file. There are also tip s on
trou bleshooting issu es w ith container view s.
Container View customization best practices
Ad d ing nod es d irectly to the cointainers.xm l is alw ays p referable to
exp and ing a p articu lar container‟s view u sing GUI op eration and then
u sing the “Save Layou t” op tion in the Container Setu p d rop d ow n
m enu . The reason for this is that any objects w hich are ad d ed to the
containers via the “Exp and ” op tions are only saved by object ID in t he
containers.xm l file. This m akes ongoing m anagem ent u sing the xm l
file d ifficu lt.
Avoid u se of the “Inclu d e N od e . . .” m enu item w hen u sing filters.
The tw o d on‟t m ix w ell. It‟s best to create filters and then strictly u se
filters for containm ent. The “Inclu d e N od e. . .” m enu is really best
u sed to test ou t a container view . For d ep loym ent, try to stick w ith
filter-based containers.
Jp eg (.jp g) files for backgrou nd im ages resize better than other im age
files typ es. It is best not to show connectivity betw een containers by
237
Fognet’s Field Guide to OpenView NNM
setting show ContainerConnectivity=“false” on all containers. There is
an exam p le in the w hitep ap er.
Don‟t p u t too m any nod es in a container. They are recalcu la ted every
tim e they are op ened . Try to cap the size arou nd 200 nod es or so. If
connectivity is not im p ortant to show in a container , set the below flag:
show N od eConnectivity=“false”
This m akes the view ‟s p erform ance im p rove greatly, so consid er
setting this for all containers. A neighbor view for connectivity from a
nod e can alw ays be lau nched to show connectivity . Connectivity
w ithin a container can be p roblem atic becau se key nod es m ay not p ass
the filter and m ay not be in the container, thu s p ossibly d egrad ing the
accu racy of the connectivity.
Make a second backu p of the containers.xm l file and call it
containers.raw .xm l. Ed it this file and rem ove all references to object
IDs. This w ill be u sefu l if the ET d atabase has to be w ip ed and re created from scratch.
Container View operations
If ru nning versions p rior to V7.53, th e below op erations are not
covered in:
$OV_DOC/ WhitePapers/ ContainerAd m inistration .pd f.
In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using
Extend ed Top ology u ser m anu al.
Delete a container:
Ed it the file: $OV_DB/ nnm et/ containers.xm l.
ovstop ovas w hen ed iting this file since it cou ld be changed by
ovas. Rem ove any containerReference tags and the entire
container tag for the container to be d eleted .
Renam e a container:
ovstop
ovas and ed it $OV_DB/ nnm et/ containers.xm l and
sim p le rep lace the nam e in the container nam e tag and in the
containerReference nam e tag.
238
Dynamic Views
Rem ove selected objects from a container:
Objects that had been saved from the GUI d o not have nod e
nam es associated w ith them in the contain er.xm l file. To rem ove
objects of this typ e, m ove the objects to be d eleted into a corner
of the screen, then “Save Layou t.” Ed it the containers.xm l file
and the d esired objects can be id entified by their X and Y
coord inates.
Scaling container icons:
To change the size of the container relativ e to the backgrou nd
grap hic, the size of the backgrou nd grap hic grap hic m u st be
increased . This can be d one in any grap hics ed iting tool su ch as
Microsoft Pictu re Manager. Rep lace the im age files in:
$OV_WWW/ htd ocs/ im ages/ backgrou nd s.
Restart ovas or call the follow ing URL to reload the new im age:
http:/ / <hostnam e>:7510/ topology/ hom e?reload Containers=true
Container View s access control
Container View s access control is only available in N N M V7.53 and
later. By d efau lt, every container is accessible to every u ser. An access
list can inclu d e a list of u sers and / or a list of roles. User -based access
lists p rovid e a w ay to grant access to ind ivid u al u sers. Users w ith the
role of ad m inistrator au tom atically have access to all containers, even
if the container has an access list. An em p ty access control list allow s
only ad m inistrators to access the container.
Users and roles are d efined as a p art of the u ser au thentication p roces s.
Before enabling container access control, enabling u ser au thentication
is requ ired . See p age 235 for m ore inform ation on Dynam ic View s
access control. To enable access control for a container, set the
enableAccessControl to “tru e” or “false” in the container d efinition
in the containers.xm l file. Then, ad d an access list sim ilar to the
follow ing exam p le:
<accessList>
<user name="myuser"/>
<role name="myrole"/>
239
Fognet’s Field Guide to OpenView NNM
</accessList>
If u ser access is configu red for containers, the access assignm ents
ap p ly to the N od e Statu s View , Interface Statu s View , and Alarm
View . In these view s, the nod es, interfaces, or alarm s a u ser can see are
d eterm ined by the total set of nod es in all the containers to w hich the
u ser has access.
240
Dynamic Views
Container View example
The exam p le below show s a container view that is p artially
containerized . That is, som e of the objects on the m ap s rep resent
containers, and som e rep resent p arts of the stand ard L3 view .
Screenshot is from an active installation at MøllerGru p p en in N orw ay,
w ith p erm ission to u se by Bjørn Asp rem .
N ode View
Su p p ort for filtering of nod e view s w as ad d ed in V7.5. This w as based ,
how ever, on legacy N N M filter d efinitions. V7.51 ad d ed su p p ort for
ET top ology filters to be u sed instead . This is m ore efficient, consistent
and scalable. If ru nning V7.51 or above, the p roced u re to sw itch to ETbased filters is fou nd in:
$OV_DOC/ WhitePapers/ nod eView _ETfilters.pd f
In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using
Extend ed Top ology u ser m anu al.
241
Fognet’s Field Guide to OpenView NNM
A nod e view that em p loys a filter can be called by u rl:
http:/ / "nnm server":7510/ topology/ nod eView ?filter=filternam e
V7.51 Interm ed iate Patch 18 fixed a p roblem w ith the URL ip Range
sp ecification, so these can now be u sed to qu alify nod e view s, e.g.:
http:/ / "nnm server":7510/ topology/ nod eView ?filter=filternam e&ipRange=10.2.*.*
Interface View
To set Interface View to not show d isabled interfaces, change the
inclu d eDisabled p aram eter from 1 to 0 in the follow ing file and restart
ovas u sing ovstart/ovstop, bu t only if ru nning versions of N N M
p reviou s to 7.53:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l
If ru nning N N M 7.53, change the inclu d eDisabled p aram eter from 1 to
0 in the follow ing file:
$OV_AS/ w ebapp s/ topology/ WEB-IN F/ servletRegistration/ 1IfaceServlet.xm l
Then ru n the follow ing scrip t to ad d you r changes to the w eb.xm l file:
$OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl
Restart ovas u sing ovstart/ovstop.
N ode Status and Interface Status View s
These view s w ere ad d ed in V7.53 to d isp lay service oriented
m anagem ent inform ation abou t nod es and interfaces su ch as w ho is
assigned to oversee the nod e, the cu rrent lifecycle state of the nod e,
and notes that can be ad d ed by an op erator.
These view s are note enabled by d efau lt and requ ire that ET and the
APA are enabled . Use the follow ing com m and s to enable or d isable
these tw o view s:
$OV_BIN/ enableN SVandISV.ovpl
$OV_BIN/ d isableN SVandISV.ovpl
242
Dynamic Views
Alarm View
Introd u ced in N N M V7.53, this tabu lar view is d esigned to rep lace the
Java Alarm Brow ser and has all the sam e fu nctionality. The Alarm
view requ ires that ET and the APA are enabled . N ote that Container
View Access Control (Page 239) ap p lies to Alarm View as w ell.
N eighbor View
In N N M V7.51 and higher, the d efau lt N eighbor View hop cou nt can
be changed . In com p lex netw orks, the d efau lt of “2” m ay create view s
that are too com p lex. To change the d efau lt, ed it the follow ing file and
change the defaultNumHops valu e and restart ovas u sing
ovstart/ovstop, bu t only if ru nning versions of N N M p reviou s to
7.53:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l
If ru nning N N M 7.53, change the defaultNumHops valu e in the
follow ing file:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ 1N eighborServlet.xm l
Then ru n the follow ing scrip t to ad d you r changes to the w eb.xm l file:
$OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl
Restart ovas u sing ovstart/ovstop:
VRRP View
Su p p ort for this view ap p eared in a p atch to V7.5 and a w hite p ap er
w as su p p lied w ith V7.51 in $OV_DOC/ WhitePap ers/ VRRP.p d f. This
d ocu m entation w as rolled into the Gu id e to Using Extend ed Top ology
in V7.53. VRRP View requ ires p u rchase of the Ad vanced Rou ting SPI.
VRRP su p p ort by N N M has the follow ing cap abilities:
• Discover virtual router groups on d evices that support RFC 2787
• RC-VRRP MIB allow s N ortel d evices to d iscover VRRP groups
• FOUN DRY-SN -IP-VRRP-MIB d iscovers virtual router groups
Sp ecial VRRP Op enView enterp rise events are generated w ith VRRP
p rotocol su p p ort and sp ecial VRRP correlators are also u sed . Check the
follow ing LRF flag to netm on to see if it is set to tru e:
m igrateH srpVirtualIP=true
243
Fognet’s Field Guide to OpenView NNM
If not, set this flag p er the LRF p roced u re on p age 5 and then verify
that any VRRP virtu al interfaces have not been ad d ed to the N N M
top ology u sing ovtopodump. Rem ove them u sing: ovtopofix –r
<router_name>.
Interp reting VRRP Grou p Statu s:
Unknown (blue)
N ormal (green)
Warning (cyan)
Minor (yellow )
Major (orange)
Critical (red )
State und eterm ined
The VRRP Group is operational
The VRRP group has an interface in the Active
State and an interface in the Standby state, but it
also has at least one interface that is not in the
Listen state.
The VRRP Group has an interface in the Active
state, but there is no interface in th e VRRP group
w hich is in the Stand by state.
Multiple interfaces in the VRRP group are in the
Active state, or m ultiple interfaces in the VRRP
group are in the Stand by state.
The group has no interface in the Active state.
OSPF View
ET‟s OSPF View p rovid es a rep resentation of an OSPF Area. By
d efau lt, it show s Area 0 (0.0.0.0). Tables show rou ters filtered by the
area, w ith basic d ata inclu d ing neighbors. OSPF View requ ires
p u rchase of the Ad vanced Rou ting SPI.
OSPF d iscovery u ses RFC 1253 or RFC 1850 MIBS. The RFC 1850 MIB
(OSPF V2) is backw ard s-com p atible w ith RFC 1253.
A seed IP ad d ress of an OSPF rou ter ru nning the MIB is requ ired to
start the d iscovery. Enter this seed rou ter in the configu ration file
below . The seed file can be configu red to d iscover via an inclu sive ru le
or an exclu sive ru le, bu t not both. Once the seed file is p op u lated , ru n
the startu p scrip t and check the log file for errors.
OSPF Disco configuration file:
OSPF Disco startup script:
Error log:
Database:
$OV_CON F/ nnm et/ Ospf.cfg
$OV_BIN/ ospfd is.ovpl
$OV_PRIV_LOG/ ospfd is.err
$OV_DB/ ospf/ ospfd is.data
Path View
Path View com p u tes p aths u sing SN MP qu eries in real tim e. Contrast
this w ith Problem Diagnosis, w hich u ses p robes and m aintains u sage
d ata over tim e (p age 167) and Access Path View , a contribu ted
ap p lication also know n as Sm art Path (p age 168.) N N M 7.53
244
Dynamic Views
introd u ced the ability to u se ET d ata to p rovid e m ore intelligent p aths.
To enable this featu re, set the valu e for restrictToET to true in the
follow ing file:
$OV_AS/ w ebapps/ topology/ WEB-IN F/ servletRegistration/ 1IPathServlet.xm l
Then ru n the follow ing scrip t to ad d you r changes to the w eb.xm l file:
$OV_WWW_REG/ d ynam icView s/ oneXm lFileCreator/ oneXm lFileCreator.ovpl
Restart ovas u sing ovstart/ovstop.
The follow ing configu ration file can be u sed to chang e the Extend ed
Top ology algorithm s for com p u ting p aths:
$OV_CON F/ nnm et/ topology/ NMActiveRoute.Conf
The V7.53 Gu id e to Using Extend ed Top ology lists the conf file
p aram eters and how they can be changed in Chap ter 6.
HSRP View
H SRP m anagem ent requ ires p u rchase of the Ad vanced Rou ting SPI.
N ote that som e u p grad e p ackages for N N M inclu d ed the Ad vanced
Rou ting SPI. H SRP m anagem ent is su p p orted only for Cisco H SRP and
H SRP EXT MIBs.
Cisco IOS‟s 12.1(14) and 12.2(13) requ ire m anu al configu ration for the
tracked interface inform ation (stand by IPs) to ap p ear in the MIB.
Old er versions au tom atically p op u late this info.
To configu re N N M to m anage H SRP rou ters see the Gu id e to Using
Extend ed Top ology. To force an ET Discovery, see the p roced u re on
p age 10. To check to see if ET d iscovered any H SRP grou p s:
UN IX:
$OV_SUPPORT/NM/ovet_topoquery \getAllHSRPs |
grep EntityName | cut –d: -f2
Wind ow s: Run: ovet_topoquery > out.txt and search out.txt for
EntityNam e, and look for the IP ad d ress on that line
Interp reting H SRP Statu s:
Unknown (blue)
N ormal (green)
Warning (cyan)
State und eterm ined
All stand by routers are available.
One IF in the Active state, and one
IF in the Stand by state, but also at least
245
Fognet’s Field Guide to OpenView NNM
Minor (yellow )
Major (orange)
Critical (red )
one that is not in the Listen state. One
or m ore stand bys are unavailable.
One IF in the Active state, no IFs in the
Standby state. No stand by available.
Multiple IFs are in Active state, or
m ultiple IFs in the Stand by state.
Routing functionality likely affected .
N o interface in the Active state.
Trou bleshooting H SRP:
The log file is $OV_PRIV_LOG/ ovet_d aH SRPSnm p .log
Managing overlapping address spaces (OAD )
Also know n as Du p IP and OAD, this cap ability is only su p p orted in
N N M versions 7.0 or higher and requ ires the AE version . In ad d ition,
extend ed top ology d iscovery m u st be enabled . OADs are d iscovered
and m onitored by the ovet_poll d aem on and OAD objects are
m aintained in the ET d atabases, bu t the APA need not be enabled to
m anage OADs. Discovered OAD nod es are not visible to any of the
N N M legacy d atabases or GUIs, and thu s are not d isp layed in ovw
m ap s. OAD configu ration and trou bleshooting is covered in Chap ter 3
of the Gu id e to Using Extend ed Top ology and in Chap ter of that gu id e
in V7.53.
The d evices in the ad d ress sp ace m u st be reachable from the N N M
server via a d evice that p rovid es Static N AT and those d evices m u st
have valid rou te to the N N M server. Statu s can be m anaged on the
overlap p ing d evices, only if ICMP an d/ or SN MP are not firew alled
along the w ay. In ord er for N N M‟s OAD to p rop erly w ork, the N AT
im p lem entation need s to be com p liant w ith the RFC's, not d ynam ic,
and not nested .
An exam p le of d u p ip .conf can be fou nd in the d irectory:
$OV_CON F/ nnm et/ d u p ip . A cu stom ized cop y of this d u p ip .con f
need s to be created for each overlap p ing ad d ress sp ace. A u niqu e
ad d ress
sp ace
nam e
d efines
a
su bd irectory
u nd er
$OV_CON F/ nnm et/ d u p ip so the configu ration file nam e for the
ad d ress sp ace nam ed “p hil” w ou ld have to be:
$OV_CON F/ nnm et/ d upip/ phil/ d upip.con f
The ovdupip com m and can be u sed to test and to qu ery the OAD
configu ration. There is no m an/ ref p age for d u p ip .conf in N N M 7.01
or 7.5, bu t the file itself has very good help . There is a m an/ ref for
ovdupip.
246
Dynamic Views
Seed files, as exp lained in the d u p ip .conf file, are necessary for p rop er
d iscovery of the OADs by ET. Once the configu ration has been
valid ated u sing ovdupip, “refresh configu ration” m u st be selected in
the OAD section of the ET configu ration GUI, then d iscovery for the
new OAD can be initiated in the d iscovery section.
Use the below com m and to sp ecify an overlap p ing ad d ress d om ain ID
(OADID) for an entity to be p olled . If this op tion is u sed then the
nod enam e or virtu alIP is assu m ed to be a p rivate IP ad d ress. The
OADID m u st be the ID and not the overlap p ing d om ain nam e. To m ap
the nam e to an OADID, p erform the com m and ovdupip -i a:
ovet_demandpoll.ovpl –r <OADID>
Any nod es listed in the OAD Configu ration file shou ld also be entered
into the netmon.nod iscover file and existing N N M-d iscovered nod es
m u st be p u rged from the IPMAP top ology for the OAD to be p rop erly
d iscovered by ET.
When the OAD is d iscovered , the OAD table d isp lay show s the nod e
nam e, the m anagem ent IP ad d ress, the p rivate IP ad d ress, the rou te
d istingu isher, and th e nod e‟s statu s. In the event brow ser, the ad d ress
sp ace nam e is p ostfixed to the m anagem ent IP ad d ress.
On Wind ow s, w hen m igrating from N N M version 7.0 to 7.5x, ru n the
follow ing com m and to u p d ate OAD zone ID‟s to the new 7.5x form at:
%OV_BIN%\migrate70to701ZoneNumber.ovpl
In N N M 7.0, d eleting an OAD (Overlap p ing Ad d ress Dom ain)
requ ired ru nning a fu ll d iscovery. In N N M 7.01, a fu ll d iscovery is no
longer necessary.
D elete an overlapping address spaces (OAD )
Follow this p roced u re to d elete an OAD called “haze” w ithou t
initiating fu ll d iscovery:
1.
2.
3.
4.
5.
6.
7.
Go to the $OV_CON F/ nnm et/ d upip/ haze d irectory
Ed it “d upip.seed ” file and com ment out all entries using a “#”
Go to the ET Config GUI
Select the “Overlapping Add ress Domains” tab
Press the “Refresh Configuration and Activate Changes”
After a m essage ind icating success, press the “Continue”
Select “haze” OAD from the table and press “Discover Zone”
247
Fognet’s Field Guide to OpenView NNM
8.
9.
10.
11.
12.
13.
14.
15.
Wait until “Please wait” m essage d isappears, then press “Continue”
Wait to see a “Discovery Com plete” pop -up m essage from Hom e Base
Go to OAD View from Hom e Base, and confirm “ haze” is not in the Table
Delete d irectory "$OV_CON F/ nnm et/ d u pip/ haze" and its contents
Go to the ET Config GUI
Select the “Overlapping Add ress Domains” tab
Press the “Refresh Configuration and Activate Changes”
After a m essage ind icating success, p ress the “Continue”
N N M 7.0, d eleting an OAD (Overlap p ing Ad d ress Dom ain) requ ired
ru nning a fu ll d iscovery. In N N M 7.01, a fu ll d iscovery is no longer
IPV6
N N M‟s IPV6 su p p ort is an ET featu re - IPV6-d iscovered d evices are
not visible u nd er legacy N N M d atabases or GUIs. IPV6 d evices are
d iscovered and p olled via ICMPv6 p ing, and IPV6 d iscovery is not
enabled by d efau lt. IPV6 d iscovery can be enabled by ru nning:
setupExtTopo.ovpl
The IPV6 d ynam ic view s in N N M 7.5 and higher show s layer 3
connectivity. Licenses are requ ired for both IPV4 (N N M) & IPV6 (ET),
and only d u al-stacked (IPV4 & IPV6) rou ters are su p p orted (i.e. t he
su p p orting rou ter m u st be d iscovered via netmon first). H itachi, N EC,
Ju nip er and Cisco equ ip m ent is su p p orted , and IPV6 d iscovery is only
su p p orted on H P-UX 11.11 and Solaris 8 & 9 N N M installations.
Only RFC 2465-su p p orted d evices can be fu lly m anaged (i.e. p rop erly
m ap p ed in top ology), bu t v6 end -nod es are d iscovered and p olled for
statu s via ICMPv6 p ing. An ad d itional license for the IPV6 nod es is
requ ired . For H P-UX, version 11i is requ ired along w ith the
ap p rop riate OS bu nd les. More info can be fou nd u nd er the IPV6 SPI
for N N M at H P's w eb site.
Configu re IPv6 d iscovery u sing the p roced u re ou tlines in the Gu id e to
Using Extend ed Top ology. There, the follow ing five control files in the
$OV_CON F/ nnm et d irectory are d ocu m ented :
IPv6.conf
IPv6Polling.conf
IPv6Prefix.conf
IPv6Scope.conf
IPv6Seed .conf
The follow ing IPV6 Statu s events are su p p orted in N N M 7.x:
- Ad d ress DOWN
248
Dynamic Views
- Ad d ress UP (log only)
- Interface Status Change (log only)
- N od e Status Change (log only)
- Prefix Group Status Change (log only)
There is a Pair-w ise correlation for Ad d ress UP and Ad d ress DOWN .
If Ad d ress UP occu rs less than 10 m inu tes after a corresp ond ing
Ad d ress DOWN , both events are su p p ressed . See p age 124 for m ore
abou t this correlation.
If an IPv6 nod e is d ow n or u nreachable d u ring N N M Extend ed
Top ology d iscovery, N N M m ay not inclu d e that nod e in the ET
d atabase (d ep end ing on the p resence of that nod e's ad d resses in the
ARP caches of IPv6-resp onsive nod es). This d iffers from the behavior
of IPv4 nod es, w hich w ill be treated as "u nresp onsive" nod es d u e to
their p resence in N N M's d atabases.
On Solaris p latform s, if IPv6 nod es are not configu red w ith hostnam es,
pmd p erform ance w ill su ffer as the m onitorIPv6Agent send s (m any)
statu s events abou t these nod es. To ad d ress this issu e, configu re the
ip N oLooku p cache to p revent these looku p s by:
1.
2.
3.
Discover IPv6 d evices
Ad d the ad d resses d iscovered to a file called <filenam e>
Run snmpnolookupconf -file <filename>
Port Admin tool
This tool introd u ced in V7.53 p rovid es a GUI for d isabling/ enabling
APA statu s p olling to p orts of interfaces that form erly requ ired u sing
the ovet_toposet.ovpl com m and . To lau nch the tool, select a target
nod e in any Dynam ic View and choose m enu selection Ed it -> ET
Monitoring Manager -> Port Ad m in Tool.
Aggregated port support
Su p p ort for aggregated p orts in ET started w ith V7.5, w here m u ltip le
p hysical p orts on d evices that su p p ort Cisco PAgP w ou ld be
rep resented via tru nk virtu al p orts. V7.51 ad d ed su p p ort for N ortel‟s
MLT and SMLT via the ovet_d aBaySSw (BaystackSw itchAgent) and
ovet_d aPassSw (Passp ortSw itch) d evice agents. For m ore inform ation
on su p p ort for N ortel aggregated p orts, see the follow ing w hite p ap er:
$OV_DOC/ Whitepapers/ MLT.pdf
249
Fognet’s Field Guide to OpenView NNM
In V7.53, this d ocu m entation is rolled u p into the Gu id e to Using
Extend ed Top ology u ser m anu al. For d etails on how aggregated p orts
are hand led by the APA, see Page 77.
Su p ort for d iscovery of aggregated p orts on Alcatel Tim etra and
Riverston sw itches w as ad d ed via new d evice agents in Ju ne, 2007. See
Page 219 for how to check for latest d evice agents.
Increase JAVA heap size
In large environm ents, increasing JAVA heap size w ill help red u ce
ru nning ou t of m em ory issu es that m ay crop u p . Ad d the follow ing
LRF flag “-Xm x384m ” to ovas.lrf p er the p roced u re on Page 5. This
sp ecifies a heap size of 384 Meg. The m ax is 512m bu t this is not
recom m end ed and the d efau lt is 128 MB.
Add, delete, manage or unmanage objects via URLs
This hand y featu re w as ad d ed in N N M 6.41 to p rovid e a rem ote
facility to rep lace loadhosts, ovtopofix and other assorted local
facilities for m anip u lating objects. The URLs are:
http:/
http:/
http:/
http:/
/
/
/
/
<N N M-Server>::7510/
<N N M-Server>::7510/
<N N M-Server>::7510/
<N N M-Server>::7510/
topology/ ad d
topology/ d elete
topology/ manage
topology/ unmanage
These URLs are configu red via the follow ing xm l files:
All: $OV_AS/ w ebapps/ topology/ WEB-IN F/ d ynam icView sUsers.xm l
Prior to NN M 7.53: $OV_AS/ w ebapps/ topology/ WEB-IN F/ w eb.xm l
7.53: …w ebapps/ topology/ WEB-IN F/ servletRegistration/ 1ManageServlets.xm l
250
Troubleshooting dynamic view s
The first p lace to look w hen trou bleshooting Dynam ic View s issu es is
the JAVA console.
On UN IX use:
On Wind ow s:
/ opt/ java1.4/ jre/ binControlPanel
Control panel -> Java Plug-in
Alw ays confirm the Web server is ru nning.
On UN IX:
On Wind ow s:
ps –ef | grep ovhttpd
Control panel ->Ad m in Tools ->
Internet Inform ation Manager ->
Web server
Confirm there is a m atch betw een top ology view s and top ology
d atabases. For m ore inform ation on ovet_top oqu ery, see p age 91. For
m ore inform ation on ovtop od u m p see p age 67.
Several p roblem s w ith d ynam ic view s m ay ar ise d u e to JAVA caching.
Caching can be safely d isable from the JAVA control p anel m entioned
above.
D ynamic view s update issues due to caching
There are several layers of caching p ossible that cou ld lead to stale
top ology inform ation (d u e to lack of u p d ate) in d ynam ic view s.
Brow ser cache: For IE, clear via Tools -> Internet Options
->General tab -> Tem porary Internet files -> Delete Files.
Proxy servers: These also often d o caching. The best practice is to d isable. For IE
d o the follow ing: Tools -> Internet Options -> Connections tab ->
LAN Settings -> Uncheck "Use a proxy server for your LAN ”.
Java Plu g-in Cache:
On Wind ow s, navigate: Start -> Control Panel -> Java Plug-in ->
Cache tab -> Clear.
On UN IX, brow se to ${JPIDIR}/ jre/ ControlPanel.htm l, w here ${JPIDIR}
is the location w here the Java Plug-in is installed . The cache can be cleared
at this point.
Web content op tim izers or internet accelerators:
251
Fognet’s Field Guide to OpenView NNM
These network d evices can hijack URL requests and serve content that is
stale. In most cases, these servers can be configured to pass either requests
from certain clients or content from certain servers.
D ynamic view s registration files
Extensions to d ynam ic view s are ad d ed u sing XML files u nd er:
$OV_WWW_REG/ d ynam icView s/ C/
H P m enu s are in d ynam icView s.xm l (d o not m od ify this file). All files
get m erged au tom atically into m enu settings.xm l located in:
$OV_WWW/ htdocs/ $LAN G/ d ynam icView s
Many trad ition al ovw m enu settings are only visible to d ynam ic view s
that are lau nched locally. Most SN MP grap hing tools u nd er the
p erform ance m enu bar fall into this category. To enable these m enu
item s to be visible to those w ho lau nch d ynam ic view s from rem ote
brow sers, change the localOnly flag from tru e to false for each m enu
item in the d ynam icView s.xm l file.
D ynamic view s and edge connectivity
Dynam ic View s has sp ecial su p p ort for d isp laying layer 3 ed ge rou ter
connectivity across WAN s in V7.5 and later. In that version, p oint-top oint connectivity is ad d ressed , bu t not p oint -to-m u lti-p oint.
Exam p les of su p p orted p rotocols inclu d e ATM, Fram e Relay, t1/ t3
and Sonet. Typ ically, su ch connections w ill be tw o -nod e su bnets w ith
30-bit su bnet m asks, and this is configu rable in the follow ing file:
$OV_CON F/ nnm et/ Ed ge3Conn.cfg
Transfer MIB application builder apps to D ynamic view s
Follow the step s below to transfer ovw Menu bar ap p lications that
w ere bu ilt u sing the MIB Ap p lication Bu ild er into Dynam ic View s:
1.
To invoke MIB App lication Build er app through a “trad itional” web interface;
use:
http:/ / nnm server:3443/ OvCgi/ snm pview er.exe
Select MIB Application Build er application, select “Display in N ew Wind ow.” In
new w indow, right click and look at Properties, e.g.:
http://nnmserver.xx.com:3443/OvCgi/webappmon.exe?ins=res
new&sel=somenode.xx.com&app=IP+Tables&act=Disk+Space&arg
252
Dynamic Views
=helpBrowser+ovw%3aHPdiskSpace+%09%09++++++appendSelectL
ist+appendSelectListToTitle+%09%09++++++headingLine+1+co
mmandTitle+%22Disk+Space%22+%09%09++++++iconName+%22Disk
+Space%22+%09%09++++++cmd+rbdf&cache=2795
2.
In m enu item s XML file, copy the URL from properties w indow and m od ify
slightly, as below :
<!-- A C T I O N S
-->
<Actions>
<action actionId=―Disk Space‖
url=―http://${localhost}${localport}/OvCgi/webappmon.exe?i
ns=resnew&amp;sel=${names}&amp;app=IP+Tables&amp;act=Disk+
Space&amp;arg=helpBrowser+ovw%3aHPdiskSpace+%09%09++++++ap
pendSelectList+appendSelectListToTitle+%09%09++++++heading
Line+1+commandTitle+%22Disk+Space%22+%09%09++++++iconName+
%22Disk+Space%22+%09%09++++++cmd+rbdf&amp;cache=2795‖/>
</Actions>
3. Define <m enu> entry to use new action .
Customize dynamic view s-based alarm brow ser
The follow ing file can be u sed to configu re the JAVA -based alarm
brow ser:
$OV_WWW/ conf/ N N M.spec
Exam p les of configu ration entry p oints inclu d e color d efinitions, filter
grou p and action d efinitions.
A u sefu l cu stom ization to this file is to d isable access to the event
correlation configu ration screens . Do this bu t com m enting ou t the
follow ing line:
CATGETS(14, 11, "Event Correlation Configuration") f.action "a_id 11";
After m aking changes to the N N M.sp ec file, stop and retart the
ovalarmsrv d aem on and refresh any op en brow ser view s.
253
22. Databases and Reporting
There are several d istinct d atabases u sed w ithin recent versions of N N M.
Som e are relational and som e are flat file d atabases. The N DBM flat file
op erational d atabases inclu d e: m ap , object, top ology, snm p Collect,
Binary Event Store.
The SN MP configu ration d atabase: ovsnm p .conf, is not consid ered an
op erational d atabase. It is com p osed of 5 N DBM d atabase files, three of
w hich hold p olling and SN MP configu ration d ata, and the other tw o are
caching d atabases. cachedb hold s a list of hostnam e to IP ad d ress
m ap p ings and IP ad d ress to hostnam e m ap p ings; nolookupdb hold s
hostnam es that w ill not be resolved to an IP Ad d ress. Problem Diagnosis
(PD), w hich has been bu nd led w ith N N M AE since version 6.4, stores
p robe d ata in a text file d atabase.
Relational d atabases u sed by N N M inclu d e the d ata w arehou se and the
Extend ed Top ology DB. The d ata w arehou se (DW) hold s exp orted trend ,
top ology, and event d ata for relational u se by the rep orting su bsystem .
This is also referred to as the “em bed d ed ” d atabase, and it u ses the Solid
DBMS OEM‟d from Solid tech. Extend ed Top ology u ses the sam e Solid
DB instance as the DW, bu t is a sep arate d atabase that is not exp osed
(the schem a is not available). The DW p art of the Solid DB can be
changed to u se Oracle (or on Wind ow s, MSSQL) instead , bu t the ET p art
cannot.
D ata w arehouse
Solid DB gets d ata via au tom atic exp ort of Events, snmpCollect and
top ology
d ata.
The
exp ort
hap p ens
throu gh
the
ovrequestd
d aem on‟s
internal
sched u ler.
The
com m and
request_list schedule show s the exact tim ing for each exp ort in
UN IX cron form at). Events get exp orted every 30 m inu tes, top ology
tw ice d aily, and trend d ata every 40 m inu tes. The exp orts can be
changed by ed iting the request.properties files in the su bd irectories
u nd er:
254
Databases and Reporting
$OV_AN ALYSIS\ ovrequestd \ requests\ C\ export
Events are exp orted to the nnm _event_d etail table, snmpCollect d ata is
exp orted to the snm p _raw _trend table. The follow ing d irectory
contains the Solid DB backu p s, w hich can be entirely rem oved if there
is a need to recover d isk sp ace:
$OV_DB/ analysis/ d efault/ backup
You can stop Solid from d oing backu p s by ed iting solid .ini in
$OV_DB/ analysis/ d efau lt and com m enting the follow ing line w ith a
sem icolon:
At=23:00 backup
After m aking changes to solid .ini, ru n ovstop ovdbcheck and then
ovstart ovdbcheck.
Solid database
Details for the d ata w arehou se‟s u se of Solid are given above. The
extend ed top ology d atabase is on the sam e instance of Solid bu t it
cannot be qu eried w ith ovdwquery. The ET d atabase schem as are not
p u blicly available. ET d ata can be qu eried , how ever, u sing:
$OV_SUPPORT/ N M/ ovet_topoquery
Solid DB d ocu m entation:
http:/ / w w w.solid tech.com / d evresources/ d ocum entation.htm l
Solid ODBC Drivers:
$OV_NEW_CONF/OVDB-RUN/conf/analysis (UNIX), or
$OV_WWW/htdocs/classes/solid (Wind ow s), or
$OV_CONF/analysis/reportTemplates/Solid/driver, or
http:/ / w w w.solid tech.com / library/ software.htm l
D ata w arehouse queries
Qu eries can be m ad e to the d ata w arehou se d ata via the ovdwquery
com m and . SQL com m and s can be d irectly entered , or p laced in a file,
and then called in w ith the –file op tion. The d efau lt u ser and
p assw ord are show n in the below exam p le qu ery:
ovdwquery -u ovdb -password ovdb -file router.query
255
Fognet’s Field Guide to OpenView NNM
The –x pwdfile <password file> w as ad d ed and the –securefile
op tion w as u p d ated in N N M V7.51 to m ake this com m and N IAP
com p liant. Thu s –u and –p assw ord op tions show n above w ere
d ep recated . See the m an/ ref p ages for u sage d etails.
The Solid DB schem a can be u nraveled by looking at the table creation
scrip ts in:
$OV_CONF/analysis/sqlScripts/tables_*
The follow ing p red efined bu ilt-in p aram eters m ay be u sed in the
SQL WH ERE clau ses:
$NOW, $BEGIN_TODAY, $BEGIN _YESTERDAY, $BEGIN_WEEK,
$BEGIN_LAST_WEEK, $BEGIN_MON TH , $BEGIN_LAST_MON TH ,
$BEGIN_YEAR, and $BEGIN_LAST_YEAR
Som e exam p les of SQL qu eries follow :
Retu rn all nod es by hostnam e:
SELECT ip_hostname FROM nnm_nodes;
Retu rn all nod es by SN MP sysnam e:
SELECT sysname FROM nnm_nodes;
Retu rn all m anaged interfaces from certain hosts:
SELECT a.snmp_ifphysaddr, b.ip_hostname,
c.ip_address
FROM nnm_objects a, nnm_nodes b, nnm_interfaces c,
nnm_objects d
WHERE c.topo_id = a.topo_id
and c.node_id = d.ovw_id
and d.topo_id = b.topo_id
and b.ip_hostname like 'S%lab'
and a.ip_status != 5;
Retu rn all m anaged interfaces m atching SN MP ifDescr:
SELECT a.snmp_ifphysaddr, c.ip_address
FROM nnm_objects a,nnm_interfaces c
WHERE c.snmp_ifdescr = 'Compaq
Ethernet/FastEthernet or Gigabit NIC'
and c.topo_id = a.topo_id;
Retu rn tod ay‟s exp orted events:
SELECT * from nnm_event_detail
WHERE event_timestamp > $BEGIN_TODAY;
Retu rn only IP ad d ressed nod es and interfaces:
SELECT n."ip_hostname", n."sysname", i."snmp_ifname",
i."ip_address"
FROM "ovdb"."nnm_nodes" n,"ovdb"."nnm_objects" o,
"ovdb"."nnm_interfaces" i
WHERE n."topo_id" = o."topo_id"
and o."ovw_id" = i."node_id"
256
Databases and Reporting
and i."ip_address" <> '0.0.0.0';
Solid database performance
In highly-scaled environm ents, there is no p erform ance ad vantage to
sw itching from Solid to Oracle. Oracle hand les sim u ltaneou s access
better than Solid , bu t overall, Solid is faster than Oracle p er H P ‟s
d ocu m entation. In highly-scaled environm ents, be su re that the Solid
DB is not on the sam e d rive as the OS. Using Raid 0 or som e other
RAID in com bination w ith m oving Solid off the OS d rive can facilitate
an increase in p erform ance of 10% or m ore.
With the release of V7.51, a w hite p ap er ap p eared on how to im p rove
Solid d atabase p erform ance in the $OV_DOC/ WhitePap ers d irectory.
It is not clear from the w hite p ap er w hether the recom m end ations
p rovid ed can ap p ly to earlier version s of N N M/ Solid . Som e of them
are know n to d o so. All of the below p aram ers are set in the follow ing
file:
$OV_DB/ analysis/ d efault/ solid .ini
For CacheSize, the w hite p ap er recom m end increasing this from the
d efau lt of 10MB to 512MB. The m inim u m is 512KB and there is no
m axim u m , bu t H P recom m end s the m axim u m not exceed the am ou nt
of p hysical m em ory.
The Thread s d efau lt is 5, w hich H P says is ad equ ate for system s w ith
one or tw o p rocessors. Increase Thread s if the N N M server has m ore
than tw o p rocessors. A com m on recom m end ation is to set thread s to
(2x (nu m CPU's) + 1). Setting ForceThread sToSystem Scop e to “yes” on
N N M Servers ru nning Solaris can d ram atically im p rove p erform ance
on that p latform only. After m aking changes to solid .ini, stop and
restart all op enview p rocesses.
D etermining the version of Solid
On UN IX:
what /opt/OV/bin/ovdbrun |grep Embedded
On Wind ow s:
find /i "embedded" "%OV_DB%\analysis\default\solmsg.*"
D atabase command utilities
Data w arehou se configu ration tools:
257
Fognet’s Field Guide to OpenView NNM
ovdbcheck
ovdbdebug
ovdwconfig.ovpl
ovdbsetup
ovdwloader
ovdwunloader
Start, stop or monitor DW
Valid ates DB connections
Configure ODBC source, security
Integrate DB into existing Oracle
Reload data into DW
Copies DW to special file or to CSV
Raw d atabase m aintenance tools:
ovcoltosql
Convert snmpCollect data to trend
ovcoldelsql
Red uce or d elete raw trend d ata
ovcolqsql
Retrieve and sum marize trend data
ovdwtopo
Retrieve topology data to DW
ovdwtrend
Export, sum and trim s trend data
(com bines ovcoltosql & ovcoldelsql)
ovdwevent
Export, trim event to/ from DW
ovdweventflt
Restrict events exported to DW
Reporter URLs
Use the follow ing URLs to access the N N M rep orting interface:
http:/ / nnm server[:port]/ OvCgi/ nnm RptPresenter.exe
http:/ / nnm server[:port]/ OvCgi/ nnm RptConfig.exe
nnmRptConfig.exe is the p rogram u sed to configu re rep orts and
nnmRptPresenter.exe is the p rogram to view w eb rep orts.
Connecting Crystal Reports to N N M (Window s)
Connecting N N M rep ort d ata to Crystal Rep orts r equ ires Crystal
Rep ort 9 Professional version or greater. To d o this, follow these step s:
1. Copy stcw3230fe.d ll to C:\ w innt\ system32 on NN M system .
2. Go to ODBC in System DSN
Datasource nam e : ovd brun
Description :
N etwork Nam e : tcpip <NN M server IP> 2690
3. Within Crystal Reports:
Select create a new connection -> ODBC (RDO)
usernam e : ovd b
passw ord : ovd b
Using Oracle for the data w arehouse
The m ost com m only asked qu estion p osed by those w ho are trying to
d ecid e w hether to u se Oracle for N N M‟s d ata w arehou se in p lace of
Solid is: “Can N N M share the sam e DB instance w ith an OVO
258
Databases and Reporting
installation?” The N N M DW can ind eed share an OVO Oracle
instance, w hich is nam ed “op enview ” by d efau lt. A com m on p roblem
w ith Oracle as DW for N N M in large-scale environm ents is that the
ovdb.nnm_event_varbinds table tend s to fill u p . To extend this table,
issu e the follow ing SQL com m and :
ALTER TABLE ovdb.nnm_event_varbinds storage maxextents
unlimited;
See the “Managing” gu id e for instru ctions on changing from Solid to
Oracle for a given p latform . In su m m ary, the follow ing com m and s are
u sed to m anage the sw itch:
ovdbsetup
ovdbcheck
ovdbdebug
On UN IX p latform s, if the Oracle SID is set to anything other than the
d efau lt “op enview ,” su bsequ ent invocations of ovdbsetup m ay
generate errors. To correct this, m ake su re the correct valu e of
$ORACLE_SID is p laced in the listener.ora file, located as listed below :
H PUX:
SOLARIS:
LIN UX:
/ etc/ listener.ora
/ var/ op t/ oracle/ listener.ora
$ORACLE_H OME/ netw ork/ ad m in/ listener.ora
N N M d oes su p p ort 64-bit Oracle d atabases. A com m ent in the Su p p ort
Matrix references that Oracle is only su p p orted in 32-bit versions on
H P-UX and Solaris. This statem ent w as rem ov ed in the N N M 7.5
Release N otes. It is not the case that the Oracle d atabase server is 64bit. Som e versions of 64-bit Oracle d id not d eliver 32-bit Oracle
d atabase d rivers. Given that N N M is 32-bit, N N M w ou ld not have a
valid d atabase d river to op erate, bu t N N M now relies u p on a d atabase
d river p rovid ed by the ODBC vend or called OVoraWire. With this
d river, N N M is assu red that a 32-bit d river exists.
OD BC D river Manager Version errors
When u sing a external DB u nd er Wind ow s, the follow ing ODBC
Driver Manager p op -u p w ind ow for a version m ism atch for variou s
ODBC u tilities m ay occu r, for exam p le:
The ODBC resource DLL
(C:\WINNT40\System32\odbcint.dll)
is a different version than the ODBC driver manager
(C:\WINNNT40\System32\ODBC32.dll).
259
Fognet’s Field Guide to OpenView NNM
If these occu r, reinstall the ODBC com p onents to ensu re p rop er
op eration by ap p lying the latest Service Pack. This m ism atched DLL
p roblem is a know n issu e w ith Wind ow s and is not u niqu e to N N M.
See Microsoft Know led ge Base article: Q170769 for m ore inform ation.
Accessing D W data directly from Microsoft Excel
Install the solid ODBC d rivers (see above) on a Wind ow s client
ru nning Excel. In the control p anel create a d ata sou rce w ith the d river
and the m achines d etails. From Excel, select: Data -> Get External Data
-> N ew Database Qu ery. Select the d ata sou rce created above and
follow the p rom p ts.
D ata w arehouse compatibility: N N M 6.0 to N N M 6.1
The ODBC interface betw een N N M and Solid changed from
N N M 6.01 to N N M 6.1. In 6.01, shared m em ory w as u sed , in 6.1
TCP/ IP is u sed . N N M 6.01 (Wind ow s) w as the last version to p rovid e
Excel tem p lates and N N M 6.1 introd u ced the w eb-based rep orting
infrastru ctu re. Excel tem p lates no longer w ork after N N M 6.1.
D isable the data w arehouse
Disabling the d ata w arehou se is not a catastrop hic o p eration. In fact, it
can greatly im p rove the p erform ance of the Solid DB in highly -scaled
environm ents. The only consequ ence to d isabling the DW is that
rep orts w ill have m issing d ata. To d isable the d ata w arehou se, ru n:
ovstop ovrequestd
ovdelobj $OV_LRF/ovrequestd.lrf
More inform ation abou t LRF files can be fou nd on p age 5. The scrip ts
that p op u late and aggregate this d ata are:
ovdwquery, ovdwevent and ovcoltosql
These scrip ts are called by ovrequestd d aem on to p op u late the d ata
w arehou se. The p op u lation of d ata into the d ata w arehou se can also be
stop p ed by com m enting ou t the at job in the follow ing file:
%OV_DV/analysis/default/solid.ini
Rebuild the data w arehouse
Ru n the below com m and on any version of N N M above v6.1 to rebu ild
the d ata w arehou se from scratch. All d ata w ill be cleared in this
p rocess, and the d efau lt tables w ill be p op u lated . This is an easy w ay
260
Databases and Reporting
to recover d isk sp ace. N ote that ru nning this com m a nd w ill stop all
Op enView p rocesses, so m ake su re u ser sessions are closed .
$OV_BIN/ovdwconfig.ovpl -type embedded –reload
Rebuild the ET D atabase
Ru n the below com m and on any version of N N M above v6.1 to rebu ild
the Extend ed Top ology d atabase, w hich is a sep arate SOLID d atabase.
Stop p ing the Op enView d aem ons w ill cau se this scrip t to fail, so leave
them ru nning:
$OV_SUPPORT/ N M/ ovet_reloadTopoDBTbls.ovpl
Re-ru n setupExtTopo.ovpl after the above com m and su ccessfu lly
com p letes.
D isabling N N M 6.2 default data collections
After version 6.2 of N N M, d efau lt rep orting configu rations w ere a lot
less consu m p tive of system and d isk resou rces. Bu t v6.2 in p articu lar
had lots of d efau lt d ata collections and rep orting enabled . To d isable
these collections and rep orts, invoke the Rep ort Config GUI either
throu gh the ovw GUI or via the u rl:
http:/ / <nnm_server>:8880/ OvCgi/ nnm RptConfig.exe
or
http:/ / <nnm_server>/ OvCgi/ nnm RptConfig.exe
Dou ble-click 'View Selected Rep orts', select a rep ort and p ress 'stop';
repeat for all of the reports listed. Exit this GUI and invoke the data
collections and thresholds GUI, either from the ovw GUI or from the
command line via xnmcollect. Select the collections that are “collecting”
by default and “suspend” them. See the procedure below for recovering
snmpCollect database disk space if desired.
Follow the p roced u re below to com p letely rem ove all the SN MP
collected d ata. N ote the com m and snmpColDump can be u sed to
selectively p are the snmpCollect d atabase as w ell, and exam p les of
those com m and are fou nd in the m an/ ref p ages:
Run: ovstop snmpCollect
Delete all files und er:
$OV_DB/ snm pCollect/
$OV_DB/ snm pCollect/ stringData/
Run: ovstart snmpCollect
261
Fognet’s Field Guide to OpenView NNM
Extend and customize router performance reports
Cisco Rou ter General Perform ance Rep orts (CPU and Free m em ory),
are lim ited in the scop e of d evices w hich this rep ort ap p lies to. N ote
that in the rep ort configu ration, the "SysObjectID"
field is
.1.3.6.1.4.1.9.1.* w hich su p p orts only som e Cisco d evices. To inclu d e
other d evices, m od ify the follow ing line in the availability.ovp l scrip t:
$report_qualifier
=
'.1.3.6.1.4.1.9.1%'\n"
"AND
node.snmp_sysobjid
LIKE
Extending the number of instances in the TopN report
To accom p lish this, change the topn_count valu e in:
$OV_CONF/analysis/global.conf
Support and contributed reporting tools
Som e u sefu l rep orts can be d erived from d ata p rod u ced by the scrip ts
in the su p p ort and contrib d irectories. The su p p ort d irectory scrip ts in
p articu lar contain som e interesting top ology d ata d u m p tools.
Su p p ort Directory:
getConnected Port.sh (UNIX)
m ineETDB.ovpl
ovet_topoconnd um p.ovpl
Dum p connected port info
Gather d evice d iscovery d ata
Dum p ad d ress and entity totals
Contrib Directory:
TopoInventory.ovpl
Trend d irectory
w riteSelList
Event Directory
Various topology reports
Various SQL for trend reports
Dum ps OVw Selections to file
Various event reports
D atabase maintenance
Use the follow ing com m and s to m aintain the d atabases that feed the
d ata w arehou se:
ovdwevent, ovdwtrend, and ovdwtopo
N ote that ovdwtrend
fu nctions.
com bines ovcoltosql
ovdwtrend -trim
ovdwevent -trim
ovdwevent –trimdetail 0
262
and
ovcoldelsql
rem ove all from SN MP tables
rem ove old est from events
rem oves all event table data
Databases and Reporting
D elete old reports
This can be d one from the rep ort configu ration GUI, or by m anu ally
d eleting files located in:
$OV_SH ARE_HTDOCS/ C/ nnm / reportPresenter/
The follow ing com m and s are u sefu l for selectively d eleting old er
rep orts on Wind ow s:
cd %OV_WWW%, then rep lace yyyym m w ith year and m onth below
and ru n the 2 follow ing com m and s :
for / f "d elim s=#" %a in ('d ir d aily.yyyym m *.* / s / b') d o d el "%a \ *.*" / q
for / f "d elim s=#" %a in ('d ir d aily.yyyym m *.* / s / b') d o rm d ir "%a" / q
If u sing som e regional setting other than EN -US, you m ight end u p
w ith several cop ies of each rep ort, and this eats d isksp ace, not t o
m ention tim e need ed to ru n ovbackup.
Rebuild D W after trim to recover space (UN IX)
This p roced u re is available in Versions 6.1 and higher:
$OV_CONTRIB/NNM/dataWarehouse/repackDb.sh
This scrip t resizes the Solid d ata w arehou se by trim m ing u nu sed sp ace
taken by the d atabase. Since the Solid d atabase never gives back any
d isk allocated w hen reaching new high -w ater m arks, this scrip t allow s
recovery of u nu sed d isk sp ace after trim m ing old d ata. This scrip t: 1)
backs u p the d atabase (ju st in case), 2) u nload s the d atabase into a flat
file, 3) rem oves and recreate an em p ty d atabase, 4) reload s the
d atabase and 5) cleans-u p . To lim it size of Solid DB in N N M V6.1 or
V6.2, see H P‟s SSO d ocu m ent nu m ber KBRC00004254 at:
openview.hp.com/ sso/ ecare/ getsupportd oc?docid=KBRC00004254
Rem ove Defau lt Rep orts:
ovstop ovrequestdovdumpevents
Delete $OV_CON F/ analysis/ requests/ C/ *
Delete $OV_AN ALYSIS/ ovrequestd / requests/ *
Delete $OV_WWW/ htd ocs/ C/ nnm / reportPresenter/ Reports/ *
ovstart -v ovrequestd
Disable Defau lt Rep orts:
Backup snmpRep.conf, ed it and com m ent all lines
Delete/ recreate Solid d ata w arehou se and ET DB:
$OV_SUPPORT/redoSolid.ovpl
263
Fognet’s Field Guide to OpenView NNM
Unload DW d ata to flat files:
ovdwunloader -file trend_output_file -trend -v
ovdwunloader -file event_output_file -event -v
Disable Solid :
Prior to N N M 6.41, Solid w as only u sed for DW op erations and
cou ld be com p letely d isabled w ithou t m u ch im p act to N N M
op erations. More recent versions u se Solid for the Extend ed
Top ology d atabase and d isabling Solid w ou ld rend er all ET
fu nctions u seless. Proced u res for com p letely d isabling Solid vary
w ith N N M version and p latform .
Maintain/ p reen w eb rep orts:
The w eb rep orts are stored in :
$OV_SH ARE_HTDOCS / C/ nnm/ reportPresenter/ Reports
Su bd irectories hold rep orts by typ e, then by tim e p eriod . If the
su bd irectory is d eleted for any p articu lar d ay, for exam p le, that d ay
sim p ly is no longer be available in the Web rep ort. Once d eleted , the
rep orts cannot be recreated .
Configu re DW, connect to ODBC:
ovdwconfig.ovpl
Resolving errors w ith database maintenance programs
If the follow ing error is seeen:
Error: "Data warehouse maintenance program (ovcoldelsql)
exited with a non-zero return code of 1 and the
following message: Data specified to trim has not been
aggregated"
This is a non-critical error that ind icates an u nsu ccessfu l d ata-d eletion
from the em bed d ed d atabase or an u nsu ccessfu l attem p t to d elete d ata
that‟s not p resent in d atabase. In m ost cases, it can be ignored .
Extreme filtering in event data exports
Only the follow ing events are exp orted by d efau lt: nod e u p , nod e
d ow n, interface u p , interface d ow n, nod e ad d ed , nod e d eleted ,
threshold violation and threshold rearm . To exp ort all alarm s, create an
em p ty file in the $OV_CON F\ analysis su b-d irectory called :
N O_EXTREME_EVEN T_FILTERIN G
Make su re that no extension is ap p end ed to this file nam e , then ru n the
follow ing com m and to enable it:
264
Databases and Reporting
ovdweventflt -n '*' -o '*' -i N
D atabase size limitations
Raw d atabases:
- The binary event store DB (BES) d efault size is 16 MB and it can grow up
to 32 MB.
- The SnmpCollect d atabase can grow w ithout bound s, and can be trim m ed
w ith the ovdwtrend or snmpColDump com m and s
- Topology, Map, and Object d atabases are bound less, but in practice rarely
grow beyond a few d ozens of MBs.
Solid DB
Solid DB can be extend ed u p to abou t 16 GB (2x8 files of 2GB). 16GB
is the d efau lt m axim u m d atabase size in V7.5.
Increase size of the binary event store (BES)
The BES is also called the event d atabase or eventdb. More inform ation
can be fou nd in the ov_event m an/ ref p ages. Do not attem p t to
d ecrease the size of eventdb, as this m ay cau se the d ata w arehou se to
fail. To increase the BES beyond the d efau lt m axim u m of 32 MB, set
the follow ing LRF sw itch (p age 5) on pmd to increase the size (in this
exam p le to 128MB):
-SOV_EVENT;b128
If the op tion to log events to trap d .log is also selected (see
p age 100), u se the “-l” (ell) op tion to increase the m axim u m size of that
file, for exam p le:
-SOV_EVENT;t;l128;b128
If red u cing the size of the BES, clear the eventd b (see below ) before
restarting Op enView p rocesses.
Clear the event database (BES)
To clear the contents of the event d atabase, issu e and ovstop
com m and , then d elete the OV_DB/ eventd b d irectory. Restart N N M
u sing ovstart. The eventd b d irectory w ill be re-created au tom atically.
D ump list of managed nodes into CSV file (UN IX only)
This is a typ ical exam p le u se of ovtopodump:
ovtopodump | awk '/NODES/, length < 2' | grep -v
Unmanaged | grep IP | awk '{print $3}' | sort –u
265
Fognet’s Field Guide to OpenView NNM
D ump a clean list of all managed nodes (UN IX only)
In the com m and below , d o not su bstitu te a hostnam e w here it says
H OSTN AME, u se that exact text:
ovtopodump -lr | grep '^HOSTNAME' |
sed -e 's/^HOSTNAME:[
]*//g' | sort
D ump interface list for all nodes (UN IX only)
for a in `ovobjprint –a ―Selection Name‖ isNode=1 | awk
‗{print $2}‘|sed –e ‗s/‖//g‘|sort`
do
print $a;
ovobjprint –a ―TopM Interface List‖ ―Selection
Name‖=$a |grep –v ―TopM Interface‖ |grep –v
―OBJECT ID‖ | grep –v ―for all objects‖
done
266
23. Distributed NNM
Mu ltip le cop ies of N N M can be set u p to share d ata and d istribu te
p olling load s in a DIM configu ration. DIM is Distribu ted Internet
Discovery and it is d escribed in the N N M Gu id e to Scalability and
Distribu tion.
A CS is an N N M Collection Station . An MS is an N N M Managem ent
Station. Any cop y of N N M can be a CS, bu t not all versions of N N M can
be an MS. Setting u p DIM requ ires m u ltip le licenses for the N N M
p rod u ct.
N ote that several im p ortant new er N N M featu res, like APA, m ay not be
fu lly su p p orted in DIM environm ents. The APA in som e cases can
elim inate the need for DIM. N ote also that if DIM is in u se on an MS and
the APA is enabled , failover p olling w ill no longer w ork. If failover
p olling is requ ired for a p articu lar architectu re on an MS, the APA
cannot be enabled .
D IM limitations
DIM requ ires the u se of netmon-based d iscovery and p olling for the
MS. APA p olling is su p p orted on CSs only.
The Starter Ed ition and N N M 250-N od e Ed ition of N N M d o not
su p p ort MS featu res.
Extend ed Top ology view s only d iscover the m anaged , local and
p rim ary top ology objects on a CS or MS; as of N N M 7.5 there is no
integration of ET u nd er DIM.
Generally, m ixed versions of N N M cannot p articip ate in DIM, bu t
there are excep tions. Typ ically, sam e versions on varying p latform s are
su p p orted .
DIM can op erate across firew alls. The p orts that need to be op en are
listed on p age 283 and there is m ore inform ation on this below .
267
Fognet’s Field Guide to OpenView NNM
D IM overlap modes
These m od es d efine the behavior of the relationship betw een CSs and
MSs:
allow Overlap :
MS m anages 2 collection station s w ith overlapping d om ains. With thi m od e,
there w ill be tw o entries in the topodb and one entry in the objdb on the
MS.
u nm anageSecond ary (u nm anage local):
MS has ow n collection d omain; secondary objects unmanaged until failover .
d eleteSecond ary (d elete local):
MS d eletes CS copies of locally m anaged objects in topod b. netmon
d iscovery is on.
Ports used by MS and CS:
This chart m ay not be accu rate for every version of N N M.
Source
Dest. Protocol
SRC Ports
DST Ports
------------------------------------------------------------------------------MS
Mgd nod es UDP
1024-65535 161 SN MP
Mgd nod es MS
UDP
1024-65535 162 SN MP
CS
MS
TCP
1024-65535 162 pmd
MS
Mgd nod es ICMP
N/ A
N/ A
Any
CS or MS TCP
8880
ovw w eb
Any
CS or MS TCP
8888
ovw w eb
Any
CS or MS TCP
9999
ovw w eb
Any
CS or MS TCP
3700
ovw w eb
Polling through firew alls using D IM
To configu re the p ort that nmdemandpoll op ens w hen listening for
incom ing d em and p oll ou tp u t from a local or rem ote collection station,
ed it the $OV_CON F\ nm d em and p oll.p orts file. Each line in the file
m ay contain a single p ort nu m ber, a com m a-d elim ited list of p ort
nu m bers, or an integer range. A valid integer range is of the form m -n,
w here m and n are valid integers, and m is less than or equ al to n.
Com m ents are d enoted by a nu m ber sign (#), and cau se the rem aind er
of the line to be ignored . Blank lines are allow ed .
Filtering
N N M filters are u sed in several p laces for d iffering N N M fu nctions,
and w ith N N M 7.0+, a second sep arate filtering facility is p rovid ed for
ET fu nctions su ch as the APA Poller. ET top ology filtering is d iscu ssed
sep arately on p age 82.
268
Distributed NNM
 Discovery filters lim it w hich objects the local topology d atabase contain s.
 Topology filters lim it w hich objects on a CS are passed to a MS.
 M ap filters lim it w hich objects are show n on user m aps.
 Failover filters specify the m ost critical nod es for w hich the MS takes
over polling control should a CS becom e unreachable from the MS.
 Important N ode filters d eterm ine w hich nod es should never be flagged as
second ary failures by netmon polling, and are d iscussed on page 58.
APA Im portant nod e filtering uses a d ifferent m ethod ology as
d escribed on page 82.
 Object filters d efine netmon-based d evices to w hich object-based polling
applies.
 Persistence filters d efine objects that are placed in m em ory for backw ard
com patibility w ith certain third party API applications by d isabling on d em and subm aps for those subm aps containing objects that pass the
filter.
 DHCP filters id entify shared or floating IP ad d resses.
ovfiltercheck, ovfiltertest and ovtopodump
These com m and s can be u sed to trou bleshoot and verify filters:
ovfiltercheck -v
ovfiltertest
ovtopodump –f <filt>
Check filter syntax
Prod uce the results of the filter and
test against the topology database
Test topology d atabase against filter
V7.51 Interm ed iate Patch 18 introd u ced the “-x” op tion to the
com m and ovtopodump. This op tion is sim ilar to ovtopodump –l,
w hich p rints a su m m ary of d ata for the local nod e, only it p rints a
su m m ary of d ata consolid ated from all collection stations.
Examples of filter expressions
H ere are som e com m only u sed exp ressions:
CiscoDevices
"Match
all
Cisco
Routers"
{
"SNMP
sysObjectID" ~ 1.3.6.1.4.1.9.* }
DevicesWith3orMoreInterfaces ―‖ { numInterfaces > 2 }
These exam p les show m ap filters that both exclu d e or inclu d e:
269
Fognet’s Field Guide to OpenView NNM
Sets {
EXCLUDEDNODELIST "Excluded Nodes"
{ "DAVE.acompany.int", "BILL.acompany.int" }
INCLUDEDNODELIST "Included Nodes"
{"JOHN.acompany.int", "FRED.acompany.int" }
}
Filters {
NetsNSegs "All networks & segments" {isNetwork ||
isSegment}
NTServer "sysObjectID for NT servers"
{ "SNMP sysObjectID" == ".1.3.6.1.4.1.311.1.1.3.1.2" }
INCLUDEDNODES "Included Nodes"
{ "IP Hostname" IN INCLUDEDNODELIST }
EXCLUDEDNODES "Excluded nodes"
{ "IP Hostname" IN EXCLUDEDNODELIST }
}
FilterExpressions {
NtServerMap "NT Managed Systems"
{ !EXCLUDEDNODES && (NetsNSegs || NTServer ||
INCLUDEDNODES)
}
}
Exclu sion filter for VLAN Interfaces. N ote u se of “| | ” vs. “&&”:
NotVlan ―Not Vlan‖ isInterface && (("Selection Name" !~
"Vl") || ("Selection Name" !~ "VLAN")) }
Using external files w ithin filters
To help sep arate lists of sou rces from m aking the filters file grow too
large, external files can be referred to w ithin the filters file. N ote that
w hile w ild card s are su p p orted in filter d efinitions d irectly w ithin the
filters file, they are not su p p orted in the external files. The exam p le
below show s filters based on external files p op u lated w ith valid N N M
nod e nam es or IP Ad d resses, one on each line:
LRouters "Set of Routers from list"
{ c\:/cfg_files/routers }
LTerminalServers "Set of Terminal Servers from list"
{ c\:/cfg_files/terminalservers }
ListNodes "All nodes from list"
{ ( "IP Hostname" in LRouters ) ||
( "IP Hostname" in LTerminalServers ) }
Filtering idiosyncracies
Filters u sing SN MP entities retu rn resu lts for the nod e entity and
cannot be u sed to p rod u ce resu lts for interface entities.
Any d efined m ap filters show u p in the H om ebase selection criteria.
This can be a p ow erfu l tool for better cu stom izing d ynam ic view s.
270
Distributed NNM
Examples of xnmtopoconf commands
These are the DIM control com m and s:
To ad d :
To d elete:
To check status:
To test:
xnmtopoconf
xnmtopoconf
xnmtopoconf
xnmtopoconf
-add collection_station
-delete collection_station
-print
-test collection_station
Remove secondary objects from the database
When d isabling DIM, second ary objects can be rem oved from the
d atabase u sing the follow ing com m and in ru nning N N M V7.51 w ith
Interm ed iate Patch 18 or greater:
ovtopofix –C -X
D IM set-up in a nutshell
On the collection station:
Ed it / etc/ snm pd .conf to establish set com m unity string
Ed it filters file to set up topology filter
Ed it ovtopm d .lrf to ad d filter using LRF process (page 5), for exam ple:
OVs_YES_START:pmd,ovwdb:O -f Routers:
OVs_WELL_BEHAVED:15:
On the m anagem ent station:
Run: xnmtopoconf -print to see if CS is know n.
If not, ad d using: xnmtopoconf -add -unmanage <CS>
Options - Configure SN MP to set the SN MP set string for CS
Test collection station : xnmtopoconf -test <CS>
Configure failover: xnmtopoconf -failover <CS>
Set status check interval: xnmtopoconf -interval [n] <CS>
Manage CS: xnmtopoconf -manage <CS>
Run: ovtopodump –RISC on the MS to monitor progress
Migrating to non-D IM w ith APA
Becau se of su bstantial im p rovem ents to scalability betw een version 6
and 7.5 on N N M, it m ay be d esirable to m igrate from m u ltip le N N M
collection stations back to a single server architectu re. The ad vantages
for d oing this are: few er N N M licenses, a single top ology to m aintain,
better analysis across p olling bou nd aries by the APA, and be tter
d ynam ic m ap s. Disad vantages are: a loss of failover ability if u sing
DIM for failover, m ore com p lex and sensitive p erform ance tu ning
tasks and non-d istribu ted p olling. N ote that if netmon-based p olling is
271
Fognet’s Field Guide to OpenView NNM
not m anaging Level 2, there m ay be a large increase in the nu m ber of
m anaged interfaces w hen sw itching to APA.
Using WEB interface w ith an MS as Management Console
The follow ing w hite p ap er in the $OV_DOC/ WhitePap er d irectory
p rovid es som e tip s abou t configu ring w eb consoles to act as a “third
tier” to the m anagem ent hierachry and thu s increase the nu m ber of
op erators w ho can access a d istribu ted N N M installation:
WebUIFrom RemoteConsoles.doc
CS view s from MS Alarm Brow ser
N N M V7.51 Interm ed iate Patch 18 introd u ced an enhancem ent to the
ovalarmsrv p rocess that allow s som e Collection Station view s to be
accessed via the Managem ent Station‟s Web Alarm brow ser by
configu ring the follow ing file:
$OV_CON F/ C/ xnmeventsExt.conf
See the follow ing White Pap er for d etails on how to configu re this:
$OV_DOC/ WhitePapers/ CrossLaunch.pd f
272
24. High Availability and Backup
N N M p rovid es the ovbackup.ovpl scrip t to p erform backu p s of the
N N M d atabases and im p ortant d ata. This scrip t calls the ovpause and
ovresume com m and s that resp ectively qu iet and aw aken the d aem ons
and force in-m em ory d ata to be w ritten ou t to the ap p rop riate files. Any
op en ovw sessions w ill receive a p op u p inform ing them that OV
d aem ons have been p au sed .
The object, top ology, snm p Collect, and m ap d atabases u se N DBM flat
file d atabases. These are “sp arse” files, and thu s there can be issu es w hen
restoring if the backu p facility u sed isn‟t sp arse file-aw are.
ovbackup.ovpl backs u p op erational and / or analytical files only. Base
installation files like binaries and configu ration files are not backed u p
by ovbackup.ovpl. Also, if ovbackup.ovpl is cu stom ized , be su re to
save cop ies of the cu stom ized file since N N M u p grad es m ay over -w rite
the file.
To p iggyback onto the ovbackup.ovpl p rocess, w rite a scrip t that cop ies
ad d itional d ata to be backed u p (as w ell as restored w ith
ovrestore.ovpl) and p lace it in the follow ing d irectory:
$OV_TMP/ ovbackup
Automated database backups
Data w arehou se backu p s occu r au tom atically by d efau lt. Daily
transaction log consolid ation also occu rs by d efau lt. If ovbackup.ovpl
is being u sed regu larly, then the DW is being backed u p tw ice. Often,
errors arise if the au tom ated DW backu p and the ovbackup.ovplbased backu p s step on each other.
To d isable au tom ated DW backu p s, m od ify the follow ing file and ad d
";" before the line that starts w ith "at":
$OV_AN ALYSIS/ d efault/ solid .ini
273
Fognet’s Field Guide to OpenView NNM
Then, rem ove old backu p s in :
$OV_DB/ analysis/ d efault/ backup
To p revent ovbackup.ovpl from backing u p the DW, ru n it w ith the –
operational com m and line op tion.
DW backu p files:
Config file:
Transaction logs:
Backups:
Error log:
$OV_DB/
$OV_DB/
$OV_DB/
$OV_DB/
analysis/
analysis/
analysis/
analysis/
d efault/
d efault/
d efault/
d efault/
solid .ini
log
backup
solerror.out
Force backu p : ovbackup.ovpl -analytical -d dest_dir
ovbackup and OpenView data protector (OmniBack)
A best p ractice is to ru n: ovbackup.ovpl before ru nning Data
Protector via cron or sched u ler. If ovbackup.ovpl is ru n as a p re-exec
u sing Data Protector, a w rap p er scrip t is requ ired to establish the
ap p rop riate environm ent variables.
ovresume times out
In higher scale d ep loym ents, the d efau lt tim eou t of 5 m inu tes can
easily be reached , p articu larly w hen there are a lot of d ata collections,
cau sing ovbackup.ovpl to fail. To increase the tim eou t, find the
system call to ovresu m e w ithin the scrip t and ad d the “ -t” op tion to
increase the tim eou t, for exam p le:
system "$OV_BIN/ovresume -v -t900 > \
$OV_TMP/ovresume.output 2>&1";
Running ovbackup.ovpl as non-root user (UN IX)
ovbackup.ovpl ru ns the ovpause and ovresume com m and s, w hich
can only be ru n by root (by d efau lt). See the p roced u re on p age 12 for
instru ctions on how to allow non -root u sers to execu te these
com m and s.
ovbackup.ovpl and D ata Warehouse interaction
When the ovbackup.ovpl com m and is execu ted , it w ill instru ct the
Em bed d ed Database server to initiate an online backu p . In ord er to
allow a roll-forw ard recovery, the d efau lt backu p sched u le m u st be
d eactivated . The p rocess for d oing this is:
274
High Availability and Backup
1.
2.
3.
Copy $OV_DB/ analysis/ d efault/ solid .ini file to
$OV_DB/ analysis/ d efault/ solid .ini.old file.
Ed it $OV_DB/ analysis/ d efault/ solid .ini file.
Com m ent out "At=<tim e> backup" entry, by inserting a ";" at the
beginning of the line. For exam ple: ;At=01:00 backup
Save $OV_DB/ analysis/ d efau lt/ solid .ini file .
4.
5.
The em bed d ed d atabase w ill now be backed u p only w hen the
ovbacku p .ovp l com m and is ru n. If u se of ovbacku p .ovp l is stop p ed in
the fu tu re, the d efau lt backu p shou ld be restored by cop ying the
solid .ini.old file back to solid .ini.
ov dbcheck daemon not starting (MS SQL)
On Wind ow s system after a reboot, if ovdbcheck fails to start it m ay be
becau se the SQL server is still recovering from the reboot and is not
available to N N M yet. Sim p ly w aiting for the DB to recover shou ld
su ffice. To avoid this issu e, alw ays try to stop N N M w ith the ovstop
com m and p rior to rebooting the server.
Backing up SN MP configuration data
The com m and to save SN MP configu ration d atabase, w hich contains
p olling and SN MP com m u nity nam e settings is:
xnmsnmpconf -export <outputfile>
Integrating N N M w ith OVO database backup
Use this w hen ru nning N N M on a server along w ith OVO (O p enView
Op erations). The d atabase backu p integration scrip ts are in
$OV_CON F/ ovbacku p / checkp oint/ op erational d irectory, and contain
both
nnm_checkpoint.ovpl
and
ito_checkpoint.sh
and
$OV_CONF/ovbackup/pre-pause (w hich contains ito_oracle.sh).
Backing up the Solid database
Solid DB grow s very large in highly-scaled environm ents u sing ET. A
Solid DB greater than 2 gigabytes in size is not u nu su al. The Solid
d atabase resid es in:
$OV_DB/ analysis/ d efault
By d efau lt, the Solid backu p p rogram is au tom atically lau nched d aily
at 11p m . So if ovbackup.ovpl is in u se, then the Solid d atabase is
u neccessarily backed u p tw ice. The au tom atic backu p is m ad e to:
275
Fognet’s Field Guide to OpenView NNM
$OV_DB/ analysis/ d efault/ backup
To d isable the au tom atic Solid backu p , p u t a sem icolon before the
follow ing line in the $OV_DB/ analysis/ d efau lt/ solid ini.file then
restart the Op enView d aem ons: At=23:00 backup
To d isable Solid backu p w hen ru nning ovbackup.ovpl, ru n instead :
ovbackup.ovpl –operational
To change the location of the Solid d atabase backu p , sp ecify a new
p ath in the solid .ini file that is relative to $OV_DB/ analysis/ d efau lt,
for exam p le:
[General]
BackupDirectory=../ ../ ../ ../ ../ ../ tm p/ solid bak
Map exports and imports
The ovw GUI Map Exp ort/ Im p ort facility p reserves IPMAP
cu stom izations m ad e for a sp ecific top ology. Legacy d atabases requ ire
occasional rebu ild ing, p articu larly in d ynam ic environm ents . Most
N N M ad m inistrators create cu stom izations to their m ap s u sing
“containerized ” realm s and cu t-and -p aste op erations.
These cu stom izations can be p reserved u sing the Map Exp ort featu re.
When restoring a m ap u sing an exp orted m ap file, the assu m p tion is
that the u nd erlying top ology d ata in the d atabases is essentially the
sam e. If it is not, m any objects m ay be p laced in the “N ew Object
H old ing Area.” Before im p orting a m ap , first m ake su re that the
d iscovery is com p lete and that any objects that w ere m anu ally
p op u lated into the d atabases via loadhosts are in p lace before
p erform ing the m ap im p ort.
Map snapshots
A m ap snap shot, w hich is also p erform ed from the ovw GUI, cap tu res
a cop y of the entire m ap d atabase, and allow s for the “freezing” of
IPMAP top ology statu s at a p articu lar p oint in tim e. Use snap shots
only for that p u rp ose: to cap tu re the statu s of the top ology at that
p articu lar m om ent in tim e. Map snap shots can also be issu ed
externally u sing the ovmapsnap com m and .
276
High Availability and Backup
ovtopodbsnapshot.ovpl
This su p p ort tool w as introd u ced in N N M V7.5 and it is u sefu l for
creating backu p s and restoration p oints for both N N M‟s ovw and ET
top ology d atabases and associated configu ration files. N ote that it
issu es a com p lete N N M d aem on shu td ow n (ovstop) w hen it is
execu ted :
$OV_SUPOPORT/NM/ovtopodbsnapshot –c <outputfile>
This tool can also be u sed effectively to to m aintain a hot-stand by
server.
N ative backup and restore commands (UN IX)
Use these w hen choosing not to u se the ovbackup.ovpl scrip t. N ote
N N M legacy d atabases are “sp arse” files, so m u st be backed u p w ith
sp arse file-aw are tools. For H P-UX, u se fbackup/frestore and for
Solaris, u se ufsdump/ufsrestore. cpio is also sp arse file-aw are.
Backup sparse file d atabases u sing cpio:
cd $OV_DB/openview ; find . | cpio –pdmux \
/dir/backupfile >> /dir/logfile 2 >> /dir/logfile
Backup and restore sparse file DB‟s using fbackup or frecover to or from
tape:
fbackup –oi $OV_DB/openview –f /dir/backupfile
frecover –x –v –o –f –s /dev/rmt/om \
-i $OV_DB/openview
D IM Configuration for a hot standby system
The follow ing im p lem entation of DIM as a hot stand by u ses the
coop erative-ind ep end ent DIM m od el d escribed in the gu id e to
Distribu tion and Scalability for N N M. An ad d itional license is
requ ired , bu t in m ost cases, a d iscou nt can be negotiated throu gh the
sales p rocess. The tw o N N M servers shou ld have the sam e visibility to
the netw ork. Generally, the hot stand by shou ld be ru nning the sa m e
OS versions and have a sim ilar hard w are configu ration as the p rim ary
system .
First, m ake su re both stations have d iscovered the sam e top ology.
Cop y the configu ration files of interest from the p rim ary to the
stand by server w hen configu ring the stand by for the first tim e. Of
p articu lar interest w ill be any netmon seed files that m ay have been
277
Fognet’s Field Guide to OpenView NNM
configu red and the netm on.noDiscover file. Also, cop y the trap d .conf
file for the p rim ary.
Create the top ology filters as follow s:
For Prim ary N N M server nam ed “Prim ary,” create filters:
Primary "" { "IP Address" ~ "1.2.3.4" }
UnManaged "" { "IP Status" ~ "Unmanaged" }
For the sam e p rim ary server, create FilterExp ression :
Topofilter ""{ !Primary && !UnManaged }
Create the sam e filters on the stand by server, su bstitu ting the nam e of
the stand by server w here it says “Prim ary” above.
Up d ate ovtopmd LRF files to call the new top ology filters on each
server, p er the d etailed p roced u re in the Scalability and Distribu tion
gu id e.
On the p rim ary server, ru n:
xnmtopoconf -add -over AllowOverlap ServerB
xnmtopoconf -failover ServerB
Set the statu s check interval on the p rim ary server u sing:
xnmtopoconf -interval <n> <node>
<n> is nu m ber of m inu tes betw een statu s checks betw een the m anager
and the collection station, and <node> is the nam e of the stand by
server. It m ay take a few hou rs for ovrepld to stabilize the tw o
top ologies.
N on-D IM configuration for a “hot standby”
A hot stand by can be configu red u sing the follow ing ou tline. An
ad d itional license is requ ired , bu t in m ost cases, a d iscou nt can be
negotiated throu gh the sales p rocess. Generally, the hot stand by
shou ld be ru nning the sam e OS versions and a sim ilar hard w are
configu ration. The tw o servers m u st be ru nning the sam e version of
N N M at the sam e p atch level. The tw o N N M servers shou ld have the
sam e visibility to the netw ork. There are actu ally several w ays this can
be accom p lished , and this m ethod is neither the best nor the m ost
efficient.
278
High Availability and Backup
Cop y configu ration files of interest from the p rim ary to the stand by
server w hen configu ring the stand by for the first tim e. Of p articu lar
interest w ill be any netmon seed files that m ay have been configu red
and the netm on.noDiscover file. Also, cop y the trap d .conf file from the
p rim ary. On the p rim ary server, exp ort the SN MP configu ration
d atabase and cop y to the stand by. Im p ort it on the stand by u sing:
$OV_BIN/xnmsnmpconf –import <file>
Allow the stand by server to d iscover the entire netw ork. In d oing so,
the stand by server w ill set itself as a trap recip ient on the m anaged
nod es. Com p are the resu lts of this server‟s d iscovery to that of the
p rim ary. Differences m ay be attribu table to netw ork visibility if the
p rim ary and stand by are on d ifferent segm ents.
Run: ovbackup.ovpl on prim ary N N M server
Copy ovbackup.ovpl results to stand by server.
Run: ovstop the run ovrestore.ovpl on stand by server.
Populate the stand by server‟s FQDN in:
$OV_DB/ openview/ ovw db/ ovserver
In essence, the id ea is to let d iscovery p olling ru n on the stand by
server, bu t not statu s p olling. To d o this, ru n:
xnmpolling –statPollOff
When u sing APA p olling instead of netmon p olling:
ovdelobj $OV_LRF/ovet_poll.lrf; ovstart ovet_poll
To enable the stand by:
xnmpolling –statPollOn
or:
ovaddobj $OV_LRF/ovet_poll
ovstart ovet_poll
The above step s p rovid e an ou tline for getting started , and they can be
inclu d ed in a nightly or w eekly set of scrip ts to keep the server‟s
synchronized . Rem em ber, that w hen sw itching from stand by back to
p rim ary, the p roced u re shou ld be reversed so that any changes that
occu rred w hile the p rim ary w as d ow n are not lost w hen the p rim ary is
brou ght back online.
279
Fognet’s Field Guide to OpenView NNM
N N M as a highly available service
N N M as a highly available ap p lication in UN IX clu sters has enjoyed
increasing su p p ort from H P, bu t w ith p roblem atic su p p ort in earlier
versions. After N N M V7.01, clu stering is su p p orted for H P‟s MC
ServiceGu ard 10.06 or greater, Su n Clu ster 2.2 or greater, and Veritas
Clu ster 2.0 or greater. N N M‟s ad d ed su p p ort in V7.01 for m igratable IP
ad d resses has enhanced N N M‟s clu sterability.
V7.51 Interm ed iate Patch 18 ad d ed su p p ort for Wind ow s Clu sters, and
a w hite p ap er d escribing its configu ra tion in the V7.53
$OV_DOC/ WhitePap ers d irectory.
N N M 6.2 (p atched ) and higher su p p orts MC ServiceGu ard and Veritas
Clu ster, bu t floating IP ad d resses m u st be u nd iscovered by p lacing
them in the netm on.noDiscover file. N N M can read the ServiceGu ard
MIBs in the cmsnmpd agent, w hich allow s N N M to sep arate floating
IPs, bu t if this agent stop s ru nning, instability cou ld resu lt. N ote that
clu stered solu tions have very sp ecific hard w are requ irem ents. MC
ServiceGu ard , for exam p le, w on‟t ru n on m ost of H P‟s w orkstation class hard w are. Sim ilarly for SUN hard w are, there are m inim u m
requ irem ents for high -sp eed channel connects.
The $OV_CON F/ ov.conf file contains configu ration settings that
p ertain to u sing N N M in a clu ster. It has its ow n m an/ ref p age.
Ad d itional N N M licenses are requ ired to ru n N N M on clu ster m em ber
nod es, bu t these licenses shou ld be available at su bstantial d iscou nts.
The choice as to w hether to achieve high availability via clu stering
technology or via im p lem enting a DIM solu tion is a hard one, and
alm ost alw ays com es d ow n m aintainability issu es rather than cost or
com p lexity issu es. A general ru le of thu m b is that if the staff is fam iliar
and com fortable w ith clu stered environm ents, clu stering is the
p referred m ethod . Likew ise, DIM shou ld be the choice for a DIMcom p etent staff. See p age 277 for an exam p le DIM configu ration that
p rovid es a highly-available N N M configu ration.
Increasing favor tow ard s d isaster recovery architectu res su ggest DIMbased solu tions, w hich lend them selves m ore easily to d isaster
recovery architectu res becau se clu stering solu tions can be p roblem atic
over great d istances.
280
High Availability and Backup
Syslog in an HA environment
If the syslog facility is to be u sed in an H A environ m ent, then
syslogTrap requ ires being installed on all nod es in a H A clu ster. This
is d u e to u p d ating files that are not u nd er the shared d isks.
Ad d itionally, it is necessary to stop syslogTrap w hen bringing d ow n
the N N M p ackage. To d o this, it is recom m end ed that the follow ing
com m and s be ad d ed to the clu ster ru n com m and s and halt com m and s:
Run com mand s:
/ opt/ OV/ bin/ OpC/ opcagt –start
H alt com m and s:
/ opt/ OV/ bin/ OpC/ opcagt -stop
/ opt/ OV/ bin/ OpC/ opcagt –kill
The ord er betw een op cagt and ovstart/ ovstop calls d oes not m atter.
N N M and multiple N IC cards
H ighly-scaled N N M installations can benefit from u sing m u ltip le
N ICS on the m anagem ent server, bu t this op tion has strict lim its,
m ostly d u e to OS p rocessing requ irem ents w ithin the netw ork stack.
N N M‟s d efau lt p oller netmon is single-thread ed , so ou tbou nd p olls are
likely to be bou nd to the sam e interface anyw ay. N N M 7.01 introd u ced
the APA p oller, w hich is m u lti-thread ed ; this p oller can take
ad vantage of m u ltip le interfaces for issu ing p olls and receiving rep lies.
For red u nd ancy, extra N ICs m ay be set u p to act as stand bys. In this
case, set u p the backu p N IC(s) w ith the sam e d efau lt rou tes as the
p rim ary N IC. Set the rou te m etric for the backu p N ICs higher than the
p rim ary, and the rou ter shou ld sim p ly u se the backu p if the p rim ary
fails or becom es too congested . Also, the USE_LOOPBACK setting in
the $OV_CON F/ ov.conf file w ill stop N N M from bind ing to a sp ecific
N IC.
Managing migratable IP addresses
Clu ster virtu al IP ad d resses that d o not fall u nd er N N M‟s ability to
hand le H SRP (see p age 245) can be hand led u sing the
netm on.m igratable configu ration file. For m ore inform ation, see the
m an/ ref p age for netm on.m igratable, w hich w as introd u ced in V6.31.
When netmon find s that an interface flagged as m igratable ap p ears on a
new d evice, it d oes not attem p t to m erge the nod es. Instead , it allow s
m ovem ent of the interface to the new d evice.
281
Fognet’s Field Guide to OpenView NNM
The IP ad d ress belonging to a m igratable interface is given a low er
p riority for u se w ithin the nod e nam ing algorithm . More inform ation
on the ord er that is u sed to d eterm ine nod e nam es can be fou nd on
p age 24.
The configu ration file accep ts single IP ad d resses, IP ad d ress ranges, or
IP ad d ress w ild card s. The file is read u p on netmon d aem on startu p , bu t
a re-read of the configu ration file is forced by ru nning:
xnmpolling –event
Trou bleshoot issu es stem m ing from netm on.m igratable by u sing the
netmon –a 115 tracem ask. See p age 67 for d etails on netmon tracing.
282
25. Firewalls and Security
Many issu es arise w hen attem p ting to m anage nod es throu gh firew alls.
There is a w hitep ap er that ship s w ith N N M on firew all d etails, bu t it is
p ossible this is ou t of d ate w ith the m ost recent versions of N N M.
N N M Ports used
Many of these p orts are only accessed via the local loop back of the
N N M server.
Used by N N M d aem ons:
SN MP requests (netmon, APA):
source port ud p 1024-65535
d est port ud p 161
SN MP replies (netmon, APA):
source port ud p 161
d est port ud p 162
SN MP traps (ovtrapd):
d est port ud p 1024-6553
source port ud p 162
ICMP requests and responses (netmon, APA)
Used by N N M Web interface:
283
ovalarmsrv :
source port tcp 2345 (V6.1-) 2953 (V6.2+)
source port tcp 2346 (V6.1 2954 (V6.2+)
ovhttpd :
source port tcp 8880 (V6.2- UNIX)
source port tcp 3443 (V6.31+ UN IX)
source port tcp 80 (Window s)
ovas :
source port tcp 7510 (V6.31+) ; 8005 (V7.01+)
ovwdb :
source port tcp 9999 (V6.1-) 2447 (V6.2+)
source port tcp 37XX, one for each jovw session,
sequentially from port 3700
Fognet’s Field Guide to OpenView NNM
Reconfiguring ports used by N N M
The p orts u sed by m ost d aem ons can be configu red via LRF sw itches.
See p age 5 for how to m od ify LRF files. Som e p orts w hich certain
d aem ons listen on, how ever, cannot be reconfigu red . N N M‟s ovtrapd
m u st listen on UDP p ort 162 and this cannot be changed . The p ort it
send trap s from (UDP 161), how ever, can be changed . N ote that even if
the trap receiver p ort cou ld be changed , every SN MP agent in the
enterp rise w ou ld also have to be reconfigu red to send trap s on a
d ifferent p ort, and it is an u nlikely feat to accom p lish for all agents.
pmd u ses TCP (not UDP) p ort 162 for talking to other cop ies of N N M,
and this p ort ap p arently cannot be changed . This is an issu e, thou gh,
only if ovrepld is ru nning, w hich it d oes only w hen u sing DIM (see
p age 267) or N N M-to-N N M event forw ard ing.
Highly-secure netw ork management scenario
Most
netw ork
m anagers
w ou ld
consid er
the
follow ing
recom m end ations extrem e, bu t som e u nu su al or ou t-of-control
environm ents call for extrem e m easu res to su ccessfu lly and secu rely
m anage them .
The follow ing scenario p rovid es su ch an exam p le:
Create a separate VLAN or managem ent network for the LAN
infrastructure.
Protect this m anagem ent network from the rest of the network via a firew all
or at a m inim um using access-lists.
Use d evice-based access lists or sim ilar technology to lim it SN MP access to
the WAN infrastructure from the m anagem ent netw ork.
Sim ilarly lim it access to the network m anagem ent servers. Consid er the use
of SN MPv3 on particularly critical and vulner able d evices. Finally, alw ays
m ake sure the appropriate version of cod e and or patches is running in the
m anaged environm ent.
N N M ICMP polls versus ping sw eep attacks
Som e firew alls rep ort p ing sw eep attacks from the N N M server.
N N M-issu ed ICMP p olls contain a string of all zeros in the p ayload ,
and the p acket is alw ays 64 bytes. This is an u nu su al p acket
configu ration and cannot be re-configu red . If the “-b” op tion is set in
the netm on.lrf file, this m ay be the sou rce of u nu su ally heavy ICMP
traffic. See the netmon m an/ ref for d etails.
284
Firewalls
Security issue w ith ICMP ping (UN IX only)
Som e IT secu rity d ep artm ents exp ress concern abou t N N M‟s statu s
p olls. N N M u ses root to execu te p ing. This is a requ irem ent as only
root u ser can op en raw sockets on UN IX. The raw socket is only op en
for a short tim e and only to send and receive 64 bytes of d ata. It only
accep ts the ICMP echo resp onse p acket com ing back.
D iscovering into subnets using N AT
N N M read s a target‟s SN MP ip .ip Ad d rTable and m ay rep ort
ad d resses that p rod u ce confu sing top ology view s and rep ort d ow n
statu s since they are only reachable w ithin the d iscovered su bnet, and
not from the N N M server. One w ay to avoid this is to block the
visibility of the offend ing IP ad d ress tables u sing SN MP cu t view s.
If seeing the non-N AT ad d resses is d esirable, som etim es these
ad d resses are treated as second ary an d not rep orted in the IP Ad d ress
table, then N N M ignores them by d efau lt. In this case, the “S” flag in
the oid _to_typ e or H Poid 2typ e files can be set to force read ing of
second ary ad d resses. See p age 7 for m ore on m od ifying these files.
H ere is an exam p le set of Cisco IOS com m and s to allow N N M to view
N AT and to exclu d e ip .ip Ad d rTable:
snmp-server view NAT-view mgmt included
snmp-server view NAT-view enterprises included
snmp-server view NAT-view ip.20 excluded snmp-server
community whatever view NAT-view rw 1
Monitoring D MZ devices
Som etim es, it is not d esirable to loosen DMZ -intranet firew all ru les to
allow netw ork m onitoring insid e a DMZ. One alternative is to set u p a
single host w ithin the DMZ to act as an SN MP p roxy. ICMP-based
p olling w ou ld be p reclu d ed , bu t if all the DMZ hosts su p p ort SN MP,
either netmon’s netm on.snm p Statu s file can be u sed (see p age 57), or
ET top ology filters in conju nction w ith APA p olling can be u sed to
control w hat d evices get p olled (see p age 82).
In environments where there is a large number of hosts in the
DMZ, setting up another copy of NNM in the DMZ and using
DIM (see page 268) is also a workable solution. Some third party
products, such as Tavve‟s eprobe (w w w .tavve.com ), are
specifically designed to gather network management data and
285
Fognet’s Field Guide to OpenView NNM
securely communicate to an internal network management
server.
Extended Topology configuration GUI passw ord
The Extend ed Top ology configu ration GUI p assw ord is in cleartext in:
$OV_AS\ w ebapps\ topology\ WEB-IN F\ d ynam icView sUsers.xm l
286
26. Product Details
This section su m m arizes new N N M version release n otes, d ifferences
betw een N N M p rod u ct bu nd les, OS-based fu nctionality d ifferences and
a su m m ary of m ajor IT m anagem ent p rod u cts in the Op enView su ite.
N N M 8i
N N M 8i (V8.00) w as released in N ovem ber, 2007 and is a com p letely
d ifferent p rod u ct that any p reviou s version of N N M. Up u ntil this
version, virtu ally every featu re ever introd u ced to N N M since version
3 has rem ained w ith the p rod u ct for backw ard com p atibility. That
changes d ram atically w ith N N M 8i. The au thor estim ates that 85% of
the cod e in N N M 8i is brand new .
There is no u p grad e p ath from p reviou s versions of N N M; it m u st be
installed on a fresh system . H P p lans to su p p ort and release new
versions of N N M 7.x in p arallel w ith N N M 8.x for at least the w hole of
2008. Migration tools for som e legacy N N M cu stom izations and som e
d atabase d ata w ill be available w ith N N M 8.10, d u e in the fall of 2008.
Most fam iliar N N M featu res and p rod u ct elem ents are totally absent
from 8i, inclu d ing the ovw GUI, all xnm ap p lications and all the
d atabases.
N N M 8i is a 64-bit ap p lication that requ ires 64-bit hard w are ru nning
64-bit OS versions and is bu ilt on a J2EE ap p lication server. It consists
of fu nctional su bsystem s in an n-tier architectu re. The m od u les are
organized arou nd a central notification bu s that p rovid es for
com m u nications betw een m od u les and , in the fu tu re, com m u nications
betw een d istribu ted cop ies of N N M 8.x.
The only m ajor su bsystem that w as p orted over from old er cod e is the
Extend ed Top ology cod e, bu t w ith a d ifferent d atabase, a d ifferent
GUI, and d ifferent configu ration entry p oints. The pmd event
su bsystem rem ains in p lace, bu t only to act as a forw ard er or receivers
of N N M events to/ from legacy cop ies of N N M. The new GUI can also
connect to legacy installations of N N M and d isp lay those p rod u ct
287
Fognet’s Field Guide to OpenView NNM
elem ents that w ere w eb-based . One cu riou s artifact from legacy N N M
is the ovspmd p rocess and the fam iliar ARF p rocess (see p age 6).
The au thor p rep ared a p resentation on N N M 8i for the 2008 H P
Softw are Universe conference in Las Vegas, bu t the p resentation w as
rejected by H P and not p resented there. It w as p resented for som e
Vivit local chap ter u ser grou p m eetings, how ever. It can be
d ow nload ed from :
w w w.fognet.com / hpsu08.pd f
N N M 8i architecture
Use the d iagram below to com p are and contrast 8i‟s architectu re w ith
p reviou s versions. N ote that it‟s com p letely d ifferent from the
architectu re d ep icted on p age 69.
288
Product Details
289
Fognet’s Field Guide to OpenView NNM
Product feature deltas by N N M version
8.01 (Release Date: 1/ 17/ 08):
AKA N N M 8i. See section on N N M 8i above.
8.00 (Release Date: 11/ 26/ 07):
AKA N N M 8i. See section on N N M 8i above.
7.53 (Release Date: 4/ 03/ 08):
Ad vanced Ed ition only:
Container View Access Control
N ode Status and Interface Status View s
Alarm View
Use ET data in Path View
Connect end nod es from another ET zone
Interface configuration change d etection in APA
Im port xnm snm pconf tim eouts & retries into ET
ET com m unity string d iscovery
Port ad m in tool in Dynam ic View s
Stand ard Ed ition & Ad van ced Ed ition:
Linux platform support (See OS section below )
H P-UX 11iv3 support (See OS section below )
Solaris Containers/ virtualization support
VMw are ESX3.0 support
Wind ow s Cluster support
JRE 1.5 support
7.52 (Release Date: ?/ 07):
This w as a lim ited release for sp ecial cu stom ers asking for
extend Linu x p latform su p p ort inclu d ing Red hat Linu x 2.6
7.51 (Release Date: 8/ 24/ 06):
Ad vanced Ed ition only:
Container View s – ovw-like view s for ET
Interface View s
Interface Discovery Filtering
Increm ental N od e Discovery for ET
Cisco Discovery Configuration
Connection Ed itor Enhancem ents
APA Im provem ents
Im proved ET Disvovery perform ance
Topology API stability im provements
Contrib App: SmartPath
Stand ard Ed ition & Ad vanced Ed ition:
N N M & OVPI integration pack
Automatic m erging of user-configurable files d uring patch updates
Default num ber of hops in Neighbor view is now configurable
Support for Std & Ent Ed itions of Wind ow s 2003 R2
Support for Tom cat v5
Support for Apache v2
Support for Solid DB v4.5
7.5 assorted p atch release featu res:
Ad vanced Ed ition only:
APA param eters add ed to load only polled objects into m emory
290
Product Details
APA param eters add ed for backoff polling for particularl d evices
APA preferred IP m anagem ent add ress param ters ad d ed
Im proved SN MP NoSuchObject hand lin g in APA
ET-based filters for Nod e View add ed
VRRP View
Expand ed ET d evice support
Stand ard Ed ition & Ad vanced Ed ition:
Java Web Start for w ind ow s-based Dynam ic View s clients
IPMAP Filter of netw ork sym bols connected to a gateway
ovtrapd can block traps based on com bo of IP ad d r‟s and event OID‟s
ovtrapd automatically unblocks blocked d evices after trap storm
ovtrapd .conf sw itch for rate of event flow
netmon sw itch to d isable d efault Anycast functionaility
netmon sw itch to d isable BGP routing tables queries
7.5 (Release Date: 8/ 12/ 04):
Ad vanced Ed ition only:
Cisco Card support show s “board” entity d etails
Trunk Support im provem ents w ith new icons
OAD Im provem ents: VLAN , H SRP, Pathview , Telnet
Rem oval of ovet_auth and ovet_d ffile
Im proved Topology Filtering
Im proved Layer Three connectivity in d ynam ic view s
N ew APA Status for Dynam ic View s: “Not Monitored”
N ew Topology Report for ET: Not SN MP Supported
N eighbor View context-sensitive VLAN View launch
Im provem ents to syslog tem plate
Support for MS SQL (N T Version)
Stand ard Ed ition & Ad vanced Ed ition:
View alarm s for selected nod es in d ynam ic view s
Support for Window s 2003 (N T Version)
Upd ated Java Brow sers, Plug-Ins, and Virtual Machines
Display layer 3 ed ge connectivity across WAN s
ovtrapd .conf suppresses trap storm s and traps from ad d r‟s
7.01 (Release Date: 1/ 14/ 04):
AE Only:
APA Poller can take over all netmon status polling
Fixes 7.0 issues w ith OID_to_sym
Fixes 7.0 issues w ith d eleting an OAD
SE Only:
Support for H P-UX on Itanium w ith H P-UX 11.23
Support for Red H at Linux Ad vanced Server 2.1
7.0 (Release Date: 10/ 8/ 03):
SE/ AE Prod u ct Stru ctu re introd u ced
AE Only:
Introd uction of syslog parser (AE UN IX versions only)
Extend ed Topology features supported on N T version
OAD Support introd uced
291
Fognet’s Field Guide to OpenView NNM
APA introd uced only for H SRP and OAD polling (Active Problem
Analyzer)
Support for filters in brid ge.noDiscover file
ET d iscovery status m oved to H om e Base
ovet_topod um p.ovpl introd uced
ET Discovery im provem ents (VLAN , autozone, etc)
SE & AE:
Im proved d ynam ic view s:
Introd uction of Hom e Base, m ulti-fram ed view s, “active tables,”
Authorization, authentication manage nod es via
Dynam icView sUsers.xm l, better scalability, many new icons,
extensible m enus, poster printing, toggle port labels, nod e, interface
d etails pages.
N ew event correlations introd uced and updated
Mapping of SN MP sysObjectID to sym bol m oved from
$OV_CON F/ oid _to_sym to $OV_CON F/ oid_to_sym_reg/
Support for 31 bit subnet m asks
netm on.m igratable introd uced
netm on.MACnoDiscover introd uced
SE Only:
Support for Red H at Linux Ad vanced Server 2.1
6.41 (ET 2.01) (Release Date: 3/ 12/ 03):
Correlation Com poser Introd uced
N ew correlations, includ ing DeDup and d ed up.conf
Cisco H SRP d iscovery, m onitoring, and 3 new H SRP View s
IPV6 d iscovery & m onitoring, includ ing 4 new IPV6 view s
Im proved Dynam ic View s:
Dynam ic View s m enu bar, N eighbor View , Sum mary View , Print, Icon
Label Control, Im proved zoom ing, signed applets, Discovery/ d evice
support for MPLS (isMPLS), IPV6 (isIPV6), BGP4 (isBGP4), H SRP
(isH SRP), OSPF (isOSPF), STP (isSTP), VRRP (isVRRP), Wireless
(isWireless), ET introd uces Zoning and conversion to Solid Database
Data w arehouse support for Oracle 9
AutoPass Licensing introd uced
Jakarta Tom cat ($OV_AS) replaces $OV_H PAS (App Server)
JRE 1.4.1 support in $OV_JRE
Data Collector im provem ents and new (suspend ed ) collections
N N M and OVPI integration (Perform ance Insight)
6.31 (ET 1.51) (Release Date: 6/ 19/ 02):
Web server port m oved from 8880 to 3443 (UNIX versions)
Major Event Correlation ch anges (N odeIf) affecting status
netmon status event change: nod e-centric to interface-centric
Web Based alarm brow ser support for xnm eventsExt.conf
Dynam ic View s introd uced w ith Stations, Internet,
N etwork, Segm ent, Path, N eighbor and Nod e View s
JRE 1.3.1 support in $OV_JRE, 1.4 on N T
Connected N od es renam ed Port-Ad d ress Mapping
6.2 (Release Date: 4/ 25/ 01):
292
Product Details
N ode View introd uced using Java Plug-in V 1.2
Cisco CDP View introd uced
Port Labeling Introd uced via Show Connection Labels
Show Path and ovtopod um p .ovpl
SN MP and object-based status polling by netmon
connectorL2Ports, nonConnectorL2Ports netmon sw itches
Discover/ d evice support for isFram eRelay, isRMON ,
isRMON2, isCDP, isATM, isDS1, isDS3, isSON ET.
Display Brid ge MIB table configur ation m enu bar item
View Connected Nod es m enu item sorted by port, VLAN
netm on.equivPorts introd uced for im proved port aggregation
Attribute-based Data Collection Using Filters
Baseline Threshold Setting using Standard Deviation based
on the data collected and stored in the data w arehouse. See
statTim eRanges.conf(4)
Collection of Ping Response Tim e and Ping Retries
Data collection throughput rate has im proved more than 10x
Out-of-the-box data collections configured
Im proved support for d evices that chan ge their SN MP
instance to interface mappings after a reconfiguration.
SN M PCollect performance via ovstatus -v snm pCollect.
ovdw query -secureFile and -force options
N ew Reports - Top ICMP Ping Response Tim e, Top ICMP
Ping Retries, Top RMON Segm ents By Octets, Top Fram e Relay
Congestion
Automatic Report Generation - Availability, and Inventory
Trend data exports and trim are now enabled by d efault
N ew , faster ECS 3.1 engine - Circuits im proved , faster
Zoom to H ighlight - H ighlight nod e using xnm events or Find Ed it:Find is
available in the Java-based N etwork Presenter.
Menu Graying in N etwork Presenter
N ew telnet, traceroute, and RMON m enu and pop -up item s
Link Managem ent Menu Item s in ovw :ATM, Fram e Relay,
DS1/ E1 Serial Line, DS3/ E3 Serial Line, SONET/ SDH, along w ith
various Perform ance, and Fault menu item s.
Tools:Unused IP Ad d resses.
OV_EventStorm event if an event storm is d etected .
N am e Service Response Tim e events:
OV_DN S_Perform Err, OV_DN S_Perform Warn
Data Collector (snm pCollect) response tim e event:
OV_DataColl_Busy
netmon perform ance via ovstatus -v netmon
ovactiond Trusted Com m and s in trusted Cm d s.conf
$OV_CON F/ trusted Cm d s.conf/ ALLOW_ALL file
SN MP Com m unity String Discovery via netm on.cm str
N ew ovw d b field : Preferred SN MP Ad d ress
Firew all Support of Traps Received Through a N AT
d evice via -u option in ov_event
ovdumpevents -t and -l options for tail and for tim e w ind ow
M Flag in H Poid 2type File for m ulti-hom ed nod es
Manuals Available as .pd f Files
ovwdb TCP port num ber change from 9999 to 2447
ovalarmsrv ,ovalarm srv_cm d : 2345 to 2953, 2346 to 2954
Obsolescence of Excel report tem plates
293
Fognet’s Field Guide to OpenView NNM
Wind ow s-only Featu res:
Installation into d irectories w ith spaces now supported . Default d ir
C:\ Program Files\ H P OpenView \ NN M\ .
Drag-and -Drop from ovw map.
Microsoft Term inal Server on Wind ow s 2000.
Tip-of-the-d ay.
Use of Winsock-2 instead of Winsock-1 libraries.
OVSH ELL and OVH IDESHELL trapd .conf Keyw ord s
N N M 6.1 (Release Date: 10/ 29/ 99):
Web Based Reporting Service via JDK 1.1
“SegRed ux” - Im proved Layout for Sw itched Environm ents
N T Support for Window s® 2000
Support of secure w eb servers (https:/ / )
Microsoft Managem ent Console (MMC) Snap -in
Im proved integration w ith H P OpenView ManageX
Software Update Menu via H elp:N N M->Patches
N N M 6.01 (Release Date: 4/ 5/ 99):
N ew capability in xnm graph to automatically configure d ata
collections for the data currently being show n on the graph via File >Configure Data Collector... m enu item .
xnmgraph and rnetstat support invocation for a specific
interface (not just a nod e).
N ew rnetstat options for nod e or interface
ovexprguru introd uced to graph statistics based on the
configuration of the particular d evice being graphed
Som e new expressions are shipped as part of m ibExpr.conf
N N M 6.0 (Release Date: 10/ 27/ 98):
Event Correlation Services (ECS) Support introd uced
Data w arehouse introd uced
Lightw eight em bed d ed database for NN M (Solid ) Support for
relational d atabases via ODBC
ovtopmd -Ro/-Ri options (Oracle/ Ingres) obsoleted
Support for exp ort of topology (inventory) data
Support for export of snmpCollect (trend ) data
Support for export of alarm (events) d ata
Excel tem plates to retrieve d ata into Excel spread sheets
Im proved Alarm Brow ser
Support for relational event d isplay of correlated alarm s Support for
m ultiple operators - when one operator an All operators see
acknow led ges/ d eletes/ reclassifies
N ew ind icator for correlated events
event brow ser (xnm events) renamed 'alarm brow ser'
Menu item s have changed from 'event' to 'alarm '.
The Binary Event Store (BES) replaces the trapd .log file
H P OpenView Java-based Web Interface introd uced :
H P OpenView Launcher, N etw ork Presenter, Web alarm brow ser,
SN MP Data Presenter
ovbackup.ovpl introd uced w ith ovpause, ovresume
Sim pler SPU-keyed licensing rep laces Nod e locks (netls)
Rem ote Console support across platform s
294
Product Details
Collection Station support via ovrepld across platform s
Automatic managem ent station/ Collection Station failover
DHCP support
DynaText Brow ser
Data Collector Enhancem ents
perform ance im provem ents via -I snmpCollect option
Multiple sysObjectId s per collection
Multiple nod e/ instance combinations per collection
Alarm generated when nod e up but SN MP request fails
Back-off d eferrals of collections w ithin the first hour
Menu Item s re-organized for platform com patibility
Repeater MIB (RFC 2108) support
SN MPv2C support in all NN M applications
Im proved status polling perform ance (connector d ow n ECS)
N ew "Map ->Export..." and "Map ->Im port..." menu item s
Registration files support platform -specific d efs (#ifd ef)
Integration and auto-d etection of H POV ManageX prod uct
Installation im provem ents
Perl 5.003 automatically installed
Apache Web server automatically installed (UN IX only)
Java Runtim e Environm ent for NN M automatically installed
ovwd b.auth and ovw .auth files (+ +) by d efault for w eb
Discovery via ping sw eep capability through load hosts -c
Support for GIF89a im ages on the m aps
ovspm d generates pop -up w hen N N M service term inates
N ew Contributed Applications:
Java-based grapher, Web-based Report Presenter,
H igh Availability Support/ Service Guard (H P-UX only)
Wind ow s only:
Support for WMI via Rem ote Power On
DMI 2.0 Support via Intel's DMI 2.0 Service Provid er version 1.10 and
DMI Explorer version 1.11
Support for pixmap format icons
Several other form erly UNIX-only features available
N N M Pre-6.0 release history:
5.02:
5.01:
5.0:
4.1:
4.0:
1/
6/
3/
3/
8/
16/ 98
23/ 97
1/ 97
96
95
3.31:
3.1:
3.0:
2.0:
1.0:
11/ 93
7/ 92
5/ 92
2/ 91
6/ 90
Product feature deltas by OS
N N M shares a com m on cod e base am ong the fou r su p p orted
op erating system s, so OS d ifferences are few , bu t here are som e:
OS-specific m enu bar item s, e.g., Registry Ed itor in Window s
and Emanate agent m etrics in UN IX, access to SAM in H P-UX, etc.
DMI Brow ser (ovcapsd) and RDMI d iscovery (IPX) is only
available in Window s version
295
Fognet’s Field Guide to OpenView NNM
UN IX versions use Apache Web Server, Wind ow s uses IIS
UN IX versions support Oracle, Wind ow s, SQL Server 2000
UN IX supports ovwperms com m and , Wind ow s d oesn‟t
Syslog feature in N N M 7.01+ not available on Wind ow s
Linux platform only supports NNM Stand ard Ed ition until V7.53
Linux: integration w ith Custom er View s not supported
Linux & Wind ow s: H igh availability integrations not supported
Linux supports only the OV_NodeIf event correlation
Linux does not support ovperm s.ovpl
N N M relies heavily on ind ivid ual OS host nam e lookup algorithm s,
w hich d iffer and can d irectly im pact performance.
UN IX submaps all persistent, Wind ow s: som e are transient
IPV6 is only supported on UN IX N N M AE installations
OS and softw are support matrix
N N M 8.01
2003 Ent x64 SP2
2003 Ent x64 R2 SP2
H P-UX 11iv3
Solaris 10 SPARC
Red Hat AS 4.0
Red Hat ES 4.0
Optional External DB‟s: Oracle 10.2.0.x
N N M 8.00
2003 Ent x64 SP2
2003 Ent x64 R2 SP2
H P-UX 11iv3
Red Hat AS 4.0
Red Hat ES 4.0
Optional External DB‟s: Oracle 10.2.0.x
N N M 7.53
2000 Pro, Srv, Ad v Srv, Term Serv, (SP4), XP Pro SP2,
2003 Std , Ent, SP1 or SP2
2003 R2 Std , Ent
H P-UX 11.0, 11.11 (11iv2), 11.23 (11iv2 PA or IT), 11.31 (11iv3 PA or IT)
N ote: H P-UX 11iv3 requires OVSN MP Emanate Agent
Solaris 2.8 2.9 2.10
Red hat Linux AS 2.1, Upd ate 1 (2.4.9-e.3), Update 2 (2.4.9-e.24)
Red H at Enterprise Linux 4, Update 2, Update 5 (64 bit)
N ote: Linux variants support NNM Starter Ed ition only
Java Plug-in: 11.4.2(Solaris 8), 1.4.2_02(H P-UX 11.0), 1.4.2_04(Red Hat
AS2.1), 1.4.2_05(H P-UX 11iv2-PA), H P-UX 11.23-IT), 1.4.2_06(Solaris
10)1.4.2_10(H P-UX 11iv3-PA&IT), 1.5(All except Wnd ow s XP)
Web Srvs: Apache, Apache Stronghold , Raven, MS Peer Web Services, MS
IIS 4.0, 5.0
Optional External DB‟s: Oracle 8.1.7(H P-UX 11.11, Red Hat AS2.1, Solaris 9,
Wind ow s 2003), 9.2.0.2(H P-UX 11.23-IT), 9.2.0.3(H P-UX 11.11, Red Hat
AS 2.1 & EL4, Solaris 9, Window s 2003), 9.2.0.6 (Solaris 10), 10gR1(HP-UX
11.0, & 11iv2-PA, Red Hat EL4, Solaris 8 & 10, Window s XP), 10gR2(H PUX 11.11 & 11.23-IT, red Hat EL4, Solaris 9 & 10, Window s 2000), MS
SQL 2000-SP3(Window s 2003), MS SQL 2005 (Wind ow s 2003)
N N M 7.51 (*** = w ith Intermediate Patch 15; **** = w ith IP 18)
296
Product Details
2000 Pro, Srv, Ad v Srv, Term Serv, (SP4), XP Pro,
2003 Std , Ent
H P-UX 11.0, 11.11 (11iv2)
Solaris 2.8 2.9 2.10
Red hat Linux Advanced Server 2.1 (SE only)
Java Plug-in: UN IX: 1.4.2(Sun/ mozilla), 1.4.2_02(H P-UX), 1.4.2_04(Linux),
1.4.2_05(H P 11.23), 1.4.2_06(Solaris 10), 1.5***, 1.5****
Wind ow s: 1.4.2, 1.5***
Web Srvs: Apache, Apache Stronghold , Raven, MS
Peer Web Services, MS IIS 4.0, 5.0
Optional DB‟s: Oracle 8.1.7(not 11.23), 9.2.0.2(N ot
Wind ow s), 9.2.0.3(not 11.23), 10gR1, 10gR2, MS
SQL 2000 (SP3)
N N M 7.5
2000 Pro, Srv, Ad v Srv, Term Serv, (SP4), XP Pro,
2003 Std , Ent
H P-UX 11.0, 11.11 (11i), (11.23 Itanium w / SE Only)
Solaris 2.8 2.9
Red hat Linux Advanced Server 2.1 (SE only)
Java Plug-in: Sun: 1.4.1(netscape), 1.4.2(m ozilla),
Wind ow s: 1.4.2, H P: 1.4.2_02, Linux: 1.4.2_04
Web Srvs: Apache, Apache Stronghold , Raven, MS
Peer Web Services, MS IIS 4.0, 5.0
Optional DB‟s: Oracle 8.1.7(not 11.23), 9.2.0.2(N ot
Wind ow s), 9.2.0.3(not 11.23), MS SQL 2000 (SP3)
N N M 7.01
2000 Pro, Srv, Ad v Srv, Term Serv, (SP1-4),XP Pro
H P-UX 11.0, 11.11 (11i), (11.23 Itanium w / SE Only)
Solaris 2.8 2.9
Red hat Linux Advanced Server 2.1
Java Plug-in: H P-UX: 1.4.1.05, others: 1.4.2_01
Web Srvs: Apache, Apache Stronghold , Raven, MS
Peer Web Services, MS IIS 4.0, 5.0
Optional DB‟s: Oracle 8.1.7(not 11.23), 9.2.0.2(N ot
Wind ow s), 9.2.0.3(not 11.23), MS SQL 2000 (SP3)
N N M 6.41
2000 Pro, Srv, Ad v Srv, Term Serv, (SP1-4*), XP Pro
H P-UX 11.0, 11.11 (11i)
Solaris 2.8 2.9
Java Plug-in: 1.4
Web Srvs: Apache, Ap ache Stronghold , Raven, MS
Peer Web Services, MS IIS 4.0, 5.0
Optional DB‟s: Oracle 8.1.7, 9.2.0.1, MS SQL 2000 (SP2)
N N M 6.31
2000 Pro, Srv, Ad v Srv, Term Serv, (SP1-3), N T 4.0 (SP6a)
H P-UX 11.0, 11.11 (11i)
Solaris 2.6, 2.7, 2.8
Java Plug-in: 1.3.1 or 1.4
Web Srvs: Apache, Apache Stronghold , Raven, MS
Peer Web Services, MS IIS 4.0, 5.0
Optional DB‟s: Oracle 8.0.6, 8.1.7, 9.0.1 , MS SQL 7.0
(SP3), 2000 (SP2)
297
Fognet’s Field Guide to OpenView NNM
N N M 6.2
2000 Pro, Srv, Ad v Srv, Term Serv, (SP1-4*), N T 4.0 (SP6a)
H P-UX 10.2, 11.0, 11.11 (11i)
Solaris 2.6, 2.7, 2.8
Java Plug-in: Varies by platform for CDP, Nod e View s only
Web Srvs: Apache, Apache Stronghold , Raven, MS
Peer Web Services, MS IIS 4.0, 5.0
Optional DB‟s: Oracle 7.3.4(10.20 only), 8.0.6, 8.1.7, MS
SQL 7.0 (SP3), 2000 (SP2)
N N M 6.1
2000 Srv, Ad v Srv, Term Serv, (SP1), N T 4.0 (SP4,5,6a)
H P-UX 10.2, 11.0
Solaris 2.51, 2.6, 2.7, 2.8 (Requires installation utilities from
ftp:/ / ovw eb.external.hp.com/ pub/ dow nload/ installs)
Web Srvs: Apache, Apache Stronghold , Raven, MS
Peer Web Services, MS IIS 2.0, 3.0, 4.0, 5.0
Optional DB‟s: Oracle 7.2.3.0, 7.3.4, 8.0.4, 8.0.5, 8.0.6,
8.1.5, 8.1.6, MS SQL 6.5, 7.0
N N M 6.01
N T 4.0 (SP3,4,5,6a)
H P-UX 10.2, 11.0
Solaris 2.51, 2.6, 2.7 (Requires installation utilities from
ftp:/ / ovw eb.external.hp.com/ pub/ dow nload/ installs)
Web Srvs: Apache, MS Peer Web Svcs, MS IIS 2.0, 3.0, 4.0
Optional DB‟s: Oracle 7.2.3.0, 7.3.4, 8.0.4, 8.0.5,
MS SQL 6.5, 7.0
N N M 6.0
N T 4.0 (SP3,4,5,6a)
H P-UX 10.2, 11.0
Solaris 2.51, 2.6 (Requires installation utilities from
ftp:/ / ovw eb.external.hp.com/ pub/ dow nload/ installs)
Web Srvs: Apache, MS Peer Web Svcs, MS IIS 2.0, 3.0, 4.0
Optional DB‟s: Oracle 7.2.3.0, 7.3.4, 8.0.4, MS SQL 6.5, 7.0
* With Patch
** Not on Window s
Standard edition vs. Advanced edition
AE includ es the Problem Diagnostics prod uct
AE Provid es Duplicate IP Support for OADs
AE Supports new APA Poller (Active Problem Analyzer)
AE includ es Ad vanced Routing SPI for OSPF, H SRP, IPV6
N o Distribution features in SE, i.e., ovrepld, xnmtopoconf, but an
SE instance can act as an N N M collection station
AE prod uct support matrix for Level 2 topology at:
openview.hp.com/ prod ucts/ nnmet/ support/ d evice_support.htm l
N ode definition
N N M cou nts nod es tow ard s the license lim it by inclu d ing:
298
Product Details
IP ad d ressable nod es w ith SN MP agents
IP ad d ressable nod es w ithout SNMP agents
Level 2 nod es d iscovered by netmon
IPX nod es on Wind ow s platform s
N odes in OAD‟s d iscovered by ET (AE only)
Any nod e that is u nm anaged (beige) is not cou nted tow ard the license
nod e cou nt. To obtain a nod e cou nts, ru n:
ovotopodump –l (ell)
A chassis w ith m u ltip le SN MP agents m ay cou nt as m u ltip le nod es.
N on-SN MP entities that share a DN S nam e are consid ered a single
nod e. Sim ilarly, if interfaces on a non -SN MP read able d evice have
d istinct DN S nam es, and N N M can‟t otherw ise d eterm ine w ith w hat
nod e they are associated , they m ay be treated as sep arate nod es. This
can u su ally be straightened by ad d ing DN S A and / or CN AME record s
for the interfaces.
Ad d itional am bigu ities m ay occu r on nod es that are assigned m u ltip le
ad d resses for a single interface or w here IP ad d resses float betw een
interfaces. It is recom m end ed to p lace su ch ad d resses in the
netm on.noDiscover file and d elete them from the top ology.
Inconsistent resp onses to SN MP m ay occu r w hen Rou nd Robin DN S is
being u sed to m ap requ ests to m u ltip le ap p lication servers.
If m anaging IPV6, the IPV6 ad d ress cou nts tow ard s the license cou nt
for that p lu g-in, and it consu m es an ad d itional nod e cou nt for the
associated IPV4 ad d ress, w hich is requ ired for each IPV6 ad d ress to be
m anaged .
The follow ing ad d itional com m and s m ay p rod u ce nod e cou nts that
conflict w ith the resu lts from the ovotopodump com m and above, bu t
these com m and s are not u sed by the licensing system in d etem ining
nod e cou nts.
ovet_topodump.ovpl –info
snmpget <NNM-server> nodeCount.0
ovstatus –v netmon (polled interfaces)
ovobjprint -S (Objects)
To p revent nod es from being d iscovered tow ard the nod e cou nt, see
the section on lim iting d iscovery on p age 39. To d iscover nod es, then
u nm anage then, so m ore nod es w ill show u p on the m ap s bu t not
cou nt tow ard s the nod e cou nt, see the section on u nm anaing objects on
p age 41.
299
Fognet’s Field Guide to OpenView NNM
N N M smart plug-ins
N N M SPI for Ad vanced Routing
H SRP, OSPF, and IPV6 Management. Requires AE and separate license,
60-day d em o installed by d efault w ith AE
N N M SPI for IP Telephony (One for Avaya and one for Cisco)
N N M SPI for LAN/ WAN Ed ge; form erly Fram e Relay SPI
N N M SPI for MPLS VPN
N N M SPI for IP Multicast
HP Softw are product suite summary
This list is lim ited to only the m ost p op u lar IT m anagem ent
ap p lications w ithin the Op enView su ite, and it d oes not inclu d e H P‟s
offerings in the storage m anagem ent, m essaging or telecom
m anagem ent sp aces.
After acqu iring Peregrine and Mercu ry in 2006 and 2007, H P
rebrand ed all its softw are to rem ove the Op enView nam e. The below
listing d oes not inclu d e all the new offerings from those verd ors, bu t
the Peregrine Service Center p rod u ct is now heavily integrated w ith
H P Service Desk offering.
Service Desk (OVSD)
H elp d esk application, ITSM/ ITIL com pliant. H P acquired Prolin in 1997
N etw ork Configu ration Manager (N CM)
Multivend or configuration and com pliance m anager. Keeps track of
configurations and changes for netw ork d evices. Checks if configurations
are com pliant w ith com pany stand ard s. OEM-ed from Voyence
Internet Services (OVIS)
SLA-oriented monitoring via softw are probes for w eb services
Transaction Analyzer (OVTA)
Monitoring and d iagnostics for both w eb and non -web J2SE, J2EE, .NET,
and COM+
Op erations (OVO)
A d istributed system s and application managem ent tool, supporting a
num ber of Sm art Plug-Ins for application m anagement (SPIs). Operations
for UN IX (OVOU) has an embed d ed copy of NN M. Operations for
Wind ow s (OVOW) does not includ e NN M. Form erly Vantage Point
Operations (VPO). Form erly IT/ Operations (ITO). Form erly Operations
Center (OPC). Form erly ManageX (acquired ManageOne 1999)
Service N avigator
Built into OVOW, and bund led w ith OVOU to provid e service-oriented
hierarchical view s. No add itional license required .
Service Inform ation Portal (SIP)
Web-based portal for serving secure view s of d ata from many OpenView
prod ucts includ ing OVOU, OVOW, N N M, OVIS, Service N avigator, OVPI,
OVPM, Reporter, OVSD and Data Protector.
Perform ance Insight (OVPI)
300
Product Details
N etwork perform ance reporting tool. Form erly Trend ; H P acquired Trinagy
in 2001
Cu stom er View s
Requires NN M; logically organizes netw ork elem ents
Canned reports for Frame, ATM, SON ET, DS-1/ E1, DS-3/ E3
Perform ance Su ite
GlancePlus: Realtim e stats for H P-UX, Solaris, AIX, & Linux
Agent: Gathers and trend s system stats, form erly MeasureWare.
Manager: Gather, d isplay d ata from agents, form erly PerfView .
Rep orter
Extensible reporting application generates pre-d efined Web reports from
Perform ance agents and OVO.
Problem Diagnostics (PD)
Software probe-based network path analysis
Bund led into NN M 7.0+ AE; no longer a separate prod uct
Event Correlation Services (ECS)
State m achine for event correlation runtim e bund led w ith NN M since 6.0;
Correlation Com poser ad d ed in 6.41 allow s d evelopm ent of ECS logic; ECS
Designer is a separate prod uct for Full ECS circuit d evelopm ent capabilities
Rou te Analytics Managem ent System (RAMS)
IP Routing analysis and trend ing probes; Separate Prod uct, works only w ith
N N M 7.5 AE; OEM‟s from Packet Design in 2004.
SN MP Research‟s SN MP Secu rity Pack.
Separate, non-H P add -on for SN MPv3 support.
Bund led w ith NN M 7.5+ (AE only).
Extensible SN MP Agent
Standalone extensible agent for HP-UX or Solaris only.
Rad ia
H P acquired Novad igm in 2004 to fill out their gap in change, configuration
and release m anagem ent.
Data Protector.
Enterprise backup application. Form erly Om niBack
Oracle for Op enView
H P-bund led Oracle database for supported OpenView apps.
PolicyXp ert
Provid es packet-level band w id th m etering and QOS man agem ent.
Insight manager integration
System s Insight, form erly Com p aq Insight Manager (CIM) Integration:
http:/ / h18013.ww w1.hp.com / prod ucts/ servers/ managem ent/ openview /
ind ex.htm l
All SIM d ow nload s:
http:/ / h18000.ww w1.hp.com / prod ucts/ servers/ managem ent/ hpsim / do
w nload .htm l
301
Fognet’s Field Guide to OpenView NNM
Core N N M product numbers (U.S), versions 6.4-7.51
Product N umber
All operating systems:
Version
D escription
J5323BA
J5324BA
T2489AA/ BA
T2490AA/ BA
T2578AA/ BA
6.4
6.4
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
Manuals, English
Med ia
Med ia
Starter Ed ition Manuals, English
Ad vanced Ed ition Manuals, Eng.
J5316BA
J5315BA
J5330BA
J5328BA
J5329BA
J1249AB
J1253UB
J5310BA
T2484AA/
T2646BA
T2688BA
T2491AA/
T2692BA
T2700AA/
T2704AA/
T2708AA/
T2712AA/
T2637AA/
T2640AA/
T2643AA/
T2651BA
T2495AA/
T2500AA/
T2505AA/
T2511AA/
T2579AA/
6.4
6.4
6.4
6.4
6.4
6.4
6.4
6.4
7.0/
7.5
7.5
7.0/
7.5/
7.0/
7.0/
7.0/
7.0/
7.0/
7.0/
7.0/
7.5/
7.0/
7.0/
7.0/
7.0/
7.0/
N N M 250 Ed ition
Enterprise Ed ition (Unlim ited )
Upgrad e N N M 250 to 6.4 250
Upgrad e N N M 250 to 6.4 Entp
Upgrad e N N M Entp to 6.4 Entp
N N M 250 Nod e Increm ent
N N M 250 Nod e Incr. Upgrad e
Developer‟s Toolkit
Starter Ed ition 250 Nod es (SE)
Starter Ed ition Unlim ited
Upgrad e SE 250 to SE Unlim ited
Upgrad e SE 250 to AE 250
Upgrad e SE Unltd to AE 1000
Upgrad e AE 250 to AE 1000
Upgrad e AE 1000 to AE 5000
Upgrad e AE 1000 to Unlim ited
Upgrad e AE 5000 to Unlim ited
Upgrad e 6.x 250 to 7.0/ 5 SE 250
Upgrad e 6.x 250 to 7.0/ 5 AE 250
Upgrd 6.x Enpr to 7.0/ 5AE 1000
Upgrd 6.x Enpr to 7.0/ 5SE Unltd
Ad vanced Ed ition 250 N od es
Ad vanced Ed ition 1000 Nod es
Ad vanced Ed ition 5000 Nod es
Ad vanced Ed ition Unlim ited
Developers Toolkit
HP-UX:
BA
BA
BA
BA
BA
BA
BA
BA
BA
BA
BA
BA
BA
BA
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
7.5x
Solaris:
J5320BA
J5317BA
J5327BA
J5397BA
J5325BA
J1256AB
J1260UB
J5311BA
T2485AA/ BA
T2647BA
T2689BA
T2492AA/ BA
302
6.4
6.4
6.4
6.4
6.4
6.4
6.4
6.4
7.0/ 7.5
7.5
7.5
7.0/ 7.5
N N M 250 Ed ition
Enterprise Ed ition (Unlim ited )
Upgrad e N N M 250 to 6.4 250
Upgrad e N N M 250 to 6.4 Entp
Upgrad e N N M Entp to 6.4 Entp
N N M 250 Nod e Increm ent
N N M 250 Nod e Incr. Upgrad e
Developer‟s Toolkit
Starter Ed ition 250 Nod es (SE)
Starter Ed ition Unlim ited
Upgrad e SE 250 to SE Unlim ited
Upgrad e SE 250 to AE 250
Product Details
T2693BA
T2701AA/
T2705AA/
T2709AA/
T2713AA/
T2638AA/
T2641AA/
T2644AA/
T2652BA
T2496AA/
T2501AA/
T2506AA/
T2512AA/
T2580AA/
7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.5/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
Upgrad e SE Unltd to AE 1000
Upgrad e AE 250 to AE 1000
Upgrad e AE 1000 to AE 5000
Upgrad e AE 1000 to Unlim ited
Upgrad e AE 5000 to Unlim ited
Upgrad e 6.x 250 to 7.0/ 5 SE 250
Upgrad e 6.x 250 to 7.0/ 5 AE 250
Upgrd 6.x Enpr to 7.0/ 5AE 1000
Upgrd 6.x Enpr to 7.0/ 5SE Unltd
Ad vanced Ed ition 250 N od es
Ad vanced Ed ition 1000 Nod es
Ad vanced Ed ition 5000 Nod es
Ad vanced Ed ition Unlim ited
Developers Toolkit
BA
BA
BA
BA
BA
6.4
6.4
6.4
6.4
6.4
6.4
6.4
6.4
7.0/ 7.5x
7.5x
7.5x
7.0/ 7.5x
7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
7.0/ 7.5x
N N M 250 Ed ition
Enterprise Ed ition (Unlim ited )
Upgrad e N N M 250 to 6.4 250
Upgrad e N N M 250 to 6.4 Entp
Upgrad e N N M Entp to 6.4 Entp
N N M 250 Nod e Increm ent
N N M 250 Nod e Incr. Upgrad e
Developer‟s Toolkit
Starter Ed ition 250 Nod es (SE)
Starter Ed ition Unlim ited
Upgrad e SE 250 to SE Unlim ited
Upgrad e SE 250 to AE 250
Upgrad e SE Unltd to AE 1000
Upgrad e AE 250 to AE 1000
Upgrad e AE 1000 to AE 5000
Upgrad e AE 1000 to Unlim ited
Upgrad e AE 5000 to Unlim ited
Upgrad e 6.x 250 to 7.0/ 5 SE 250
Upgrad e 6.x 250 to 7.0/ 5 AE 250
Upgrd 6.x Enpr to 7.0/ 5AE 1000
Upgrd 6.x Enpr to 7.0/ 5SE Unltd
Ad vanced Ed ition 250 N od es
Ad vanced Ed ition 1000 Nod es
Ad vanced Ed ition 5000 Nod es
Ad vanced Ed ition Unlim ited
Developers Toolkit
BA
BA
BA
BA
7.5x
7.5x
7.5x
7.5x
Starter Ed ition 250 Nod es
Starter Ed ition Unlim ited
Upgrad e SE 250 to SE Unlim ited
Developers Toolkit
BA
BA
BA
BA
BA
BA
BA
BA
BA
BA
BA
BA
Window s:
J5321BA
J5321BA
J5314BA
J5398BA
J5326BA
J1242AB
J1242UB
J5312BA
T2486AA/
T2648BA
T2690BA
T2493AA/
T2694BA
T2702AA/
T2706AA/
T2710AA/
T2714AA/
T2639AA/
T2642AA/
T2645AA/
T2653BA
T2497AA/
T2502AA/
T2507AA/
T2513AA/
T2581AA/
BA
BA
BA
BA
BA
BA
BA
BA
BA
Linux:
T2487AA/
T2649AA/
T2691AA/
T2582AA/
N otes: Prod u ct nu m bers end ing in AA refer to 7.0/ 7.01 versions and
those end ing in BA refer to version 7.51 and 7.53. Prod u ct nu m bers for
N N M 6.4 and 6.41 are the sam e and p rod u ct nu m bers for N N M 7.0 and
N N M 7.01 are the sam e.
303
Fognet’s Field Guide to OpenView NNM
Support package numbers (N orth America only)
A new su p p ort stru ctu re w as introd u ced at the end of 2003. Several
com p anies d eliver SPI-su p p ort on behalf of H P, and have ad d -ons to
the su p p ort p ackages from H P. Those are not listed here.
Old
Support
Options
OS2, OS3
3Y2, 3Y3
OS6
3Y6
OS3
3Y3
OS6
3Y6
n/ a
n/ a
OTx
n/ a
U5008AA
005
012
025
304
N ew
H P Care
Pack #s
H A106A1
H A106A3
H A107A1
H A107A3
H A109A1
H A109A3
H A110A1
H A110A3
H A111A1
H A111A3
H A112A1
H A112A3
H A287A1
n/ a
n/ a
n/ a
Description
Software Sppt – Business hrs, upd ates, 1 yr
Software Sppt - Business hrs, upd ates, 3 yrs
Software Sppt – 24x7, updates, 1 yr
Software Sppt – 24x7, updates, 3 yrs
SW & H W sppt - Business hrs, upd ates, 1 yr
SW & H W sppt - Business hrs, upd ates, 3yrs
SW & H W sppt – 24x7, upd ates, 1 yr
SW & H W sppt - 24x7, updates, 3yrs
Old PSS (Personalized system sppt) – 1 yr
Old PSS (Personalized system sppt) – 3 yrs
Old CSS (Critical System s sppt) – 1 yr
Old CSS (Critical System s sppt) – 3 yrs
OpenView Prem ier Services
U2461AA Developer Support 5-pack
U2461AA Developer Support 12-pack
U2461AA Developer Support 25-pack
Glossary
A
ACL
AE
AIX
Alarm
Alp ha
AMI
AN SI
APA
ARF
ARP
ASN .1
ATM
B8ZS
BECN
BER
BES
BGP
BIN D
BRI
BSD
CA
CERT
CCO
CDC
CDP
CFA
CGI
CISC
CIDR
CIM
CIR
CLI
CN AME
DN S A Record or Ad d ress Record p er RFC 1035
Access Control List, sets access rights (MS, Cisco, etc)
Acronym for H P Op enView N N M Ad vanced Ed ition
IBM‟s p rop rietary UN IX system V variant
An N N M d ep iction of an event conveying relations, etc.
A 64-bit chip architectu re introd u ced by DEC in 1992
Alternate Mark Inversion, an old er DDS T1 encod ing
Am erican N ational Stand ard s Institu te, w w w .ansi.org
Active Problem Analyzer, N N M‟s ET-based Poller
N N M Ap p lication Registration Files, p age 6
Ad d ress Resolu tion Protocol p er RFCs 826, 1293
Abstract Syntax N otation One, an OSI-d efined syntax
Asynchronou s Transfer Mod e p er RFC 1932
Binary 8 Zero Su bstitu tion, Clear Channel T1 cod ing
A FR Backw ard Exp licit Congestion N otification
Basic Encod ing Ru les are ASN .1 encod ing ru les
Binary Event Store, N N M‟s raw event store
Bord er Gatew ay Protocol, an EGP p er RFC 1771
Berkeley Internet N am e Dom ain, DN S, w w w .isc.org
Basic Rate Interface, a typ e of ISDN interface
Berkeley Softw are Distribu tion, w w w .bsd .org
Com p u ter Associates, Inc. w w w .ca.com
Com p u ter Em ergency Resp onse Team , w w w .cert.org
Cisco Connection Online – Cisco registered su p p ort site
Cisco Discovery Configu ration, controls ET d iscovery p aram s
Cisco Discovery Protocol, ad vertises Cisco d evice d ata
N N M‟s Connectivity Fau lt Analyzer, an APA su bsystem
Com m on Gatew ay Interface for Web, RFC 3875
Com p lex Instru ction Set Com p u ting CPU architectu re
Classless Inter-Dom ain Rou ting, RFC 1518, 1519, 1817
Com p aq Insight Manager, now called SIM
Com m itted Inform ation Rate, see Fram e Relay
Com m and Line Interface, accesses OS com m and s
DN S Canonical N am e Record (Alias), RFC 1035
305
Fognet’s Field Guide to OpenView NNM
CS
CSOV
CSV
CW2K
CWSI
DB
DBM
DBMS
DDS
DE
DEC
DFM
DIM
DOD
DOS
DMI
DMTF
DMZ
DN S
DP
DS1, DS3
Du p lex
DW
E1/ E3
ECS
EGP
Envvars
Event
ET
Ethernet
FDB
FECN
FQDN
Fram e
FR
grep
GN U
GPL
GUI
HC
HP
H P-UX
H POV
306
N N M Collection Station: a DIM term
Chip Su tton‟s OV Perl API Mod u le, w w w .cs-net.com
Com m a Sep arated Variable text file for DB im p orts
CiscoWorks 2000, Cisco‟s rou ter m anagem ent tool
CiscoWorks for Sw itched Internetw orks; p art of CW2K
Generic term for Data Base
UCB‟s flat file d atabase libraries; em bed d ed in UN IX
Database Managem ent System , e.g. Oracle, SQL Server
Datap hone Digital Service, AT&T‟s p re-T1 link service
Discard Eligible Fram e Relay p acket
Digital Equ ip m ent Corp . A d efu nct com p u ter com p any
CiscoWork‟s Device Fau lt Manger, su bset of SMARTS
N N M Distribu ted Internet Discovery
United States Dep artm ent of Defense
Denial of Service attack or MS Disk Op erating System
Desktop Managem ent Interface, a DMTF stand ard
Desktop Managem ent Task Force, a stand ard s bod y
Dem ilitarized Zone betw een internet and intranet
Dom ain N am ing Service; Many RFCs, p rim arily 1035
H P Op enView Data Protector; enterp rise backu p tool
1.544Mbit(t1), 43Mbit(t3) serial lines, RFC 1232, 2495
Synonym ou s w ith Fu ll Du p lex; bi-d irectional traffic
Op enView N N M d ata w arehou se, a relational DB
See DS1/ DS3
Op enView Event Correlation Services p rod u ct
Exterior Rou ting Protocol, generic for internet rou ting
Contraction of environm ent variables
N N M-encap su lated trap con veying statu s, sou rce, etc
Extend ed Top ology, N N M‟s Level 2 top ology tool
A LAN architectu re d efined by IEEE 802.3 stand ard
Acronym for Brid ge Fow ard ing Database,
A FR Forw ard Exp licit Congestion N otification
Acronym for Fu lly Qu alified Dom ain N am e
The OSI Layer 2 (d ata link) encap su lation
Fram e Relay, WAN p acket sw itching, RFC 1315, 2115
UN IX com m and to search for a p attern
Short for GN U‟s N ot UN IX, a free OS, w w w .fsf.org
Short for GN U Pu blic License, w w w .gnu .org
also UI, an acronym for Grap hical User Interface
H igh Cap acity MIB variables su p p ort 64-bits cou nters
H ew lett-Packard Corp ., a large com p u ter com p any
H P‟s p rop rietary UN IX Sytem V variant
H ew lett-Packard Op enView , synonym ou s w ith OV
Glossary
H SRP
IAN A
IBM
ICMP
IGP
IEEE
IE
If
IfMIB
IIS
IOS
IETF
IP
IPV4
IPV6
ISO
ISDN
IT
IT
ITRC
J2EE
Java
JPI
JRE
JVM
Kerm it
LAN
LRF
MAC
Man
MAU
MD5
MIB
m ib2
MibExp r
MLT
Motif
MS
MTU
N AT
N DBM
N DE
N etbios
N etBT
H ot-Sw ap p able Rou ting Protocol, A Cisco p rotocol
Internet Assigned N u m bers Au thority, w w w .iana.org
Int‟l Bu siness Machines, a large com p u ter com p any
Internet Control Message Protocol, Packet IP, RFC 792
Interior Gatew ay Protocol, generic for intranet rou ting
Int‟l Institu te of Electronic Engineers, w w w .ieee.org
Microsoft‟s Internet Exp lorer w eb brow ser
Contraction for Interface, see N IC
Interface Mib, RFC 1573, 2233
Microsoft Internet Inform ation Server, a w eb server
Internal Op erating System ; Cisco et al.
Internet Engineering Task Force, w w w .ietf.org, see RFC
Internet Protocol, See RFCs 791, 793, 1180, etc
Internet Protocol Version 4, 32-bit IP ad d r, RFC 791
Internet Protocol Version 6, 128-bit IP ad d r, RFC 2460
International Org. for Stand ard ization, w w w .iso.org
Integrated Services Disgital N etw rok; a telep hony netw ork
Acronym for Inform ation Technology
Short for Itaniu m in OS softw are su p p ort m atrix (p age 296)
H P‟s IT Resou rce Center; online d b at w w w .itrc.hp .com
Java 2 Platform Enterp rise Ed ition, d ev p latform from Su n
Su n‟s high-level object-oriented p rogram m ing langu age
Java Plu g-In, equ ivalent to JRE
Java Ru ntim e Environm ent, http :/ / java.su n.com / j2se/
Java Virtu al Machine, a virtu alized JAVA p rocessor
Serial com m u nications from w w w .colu m bia.ed u / kerm it
Acronym for Local Area N etw ork
N N M Local Registration File
Med ia Access Control Ad d ress, AN SI/ IEEE Std 802
UN IX Man p ages p rovid e online help facilities
Med ia Access Unit, e.g an Ethernet transceiver
Message Digest Encryp tion algorithm , RFC 1321
SN MP Managem ent Inform ation Base, p er RFC 1212
MIB hold ing the m ost com m on agent d ata, RFC 1213
N N M Mib Exp ression, com bines variables u sing RPN
Mu lti-Link tru nking, an aggregated p ort p rotocol for N ortel
OSF GUI gu id elines for look and feel, op engrou p .org
N N M Managem ent Station (DIM); also: Microsoft
Maxim u m Transm ission Unit. A p acket size lim itation
N etw ork Ad d ress Translation, RFC 1631 et al.
A flat file DB library set u sed by N N M based on DBM
Cisco‟s N etflow Data Exp ort, a sw itch traffic m onitor
N etw ork Basic Inp u t/ Ou tp u t System , A Microsoft API
N etbios over TCP/ IP, an old er Wind ow s service
307
Fognet’s Field Guide to OpenView NNM
N etm ask
netmon
N etView
N IAP
N IC
N IS
N od e
N MS
NNM
N ortel
OAD
Object
Octet
ODBC
OEM
OID
OOID
Op enSSH
Op enView
OQL
OUSPG
OV
OVO
OVOW
OVPI
OVR
ovw
OS
OSF
OSF/ 1
OSI
OSI
OSPF
Packet
PA
PAgP
Perl
PID
p ing
PD
PDU
PPP
PPTP
308
Contraction of IP Su bnet netw ork m ask, RFC 950
N N M‟s d iscovery and p olling d aem on
An IBM N MS based in p art on Op enView cod e
A US Govt IT secu rityt initiative, see niap -ccevs.org
N etw ork Interface Card , a m ed ia access d evice
N etw ork Inform ation Service (Yellow p ages) (Su n)
A system or d evice connected to a netw ork
A generic term for N etw ork Managem ent System
H ew lett-Packard ‟s Op enView N etw ork N od e Manager
A large p rovid er of netw ork equ ip m ent
Overlap p ing Ad d ress Dom ain (TCP/ IP)
In N N M, a u niqu ely id entified object d atabase entity
A single byte of d ata, or 8 bits
Op en Data Base Connectivity; SQL Access Grou p Std
Original Equ ip m ent Manu factu rer‟s em bed d ed p rod u ct
Object ID, d otted nu m ber for SN MP MIB variable
Op enView Object ID, u niqu e no. assigned in object d b
Secu re op en sou rce logon tool from w w w .op enssh.com
H P‟s su ite of enterp rise m anagem ent softw are p rod u cts
A su bset of SQL w ith H P extensions for N N M/ ET
Ou lu Univ. Secu re Program m ing Grou p , ee.ou lu .fi/ ?en
A contraction of Op enView , a su ite of H P softw are
Op enView Op erations for UN IX (A system s m anager)
Op enView Op erations for Wind ow s
Op enView Perform ance Insight (form erly Trend Micro)
Op enView Rep orter Prod u ct
OV Wind ow s, the N N M static Map foregrou nd p rocess
Op erating System , e.g. Solaris, Wind ow s 2000, H P-UX
Op en Softw are Fou nd ation, now w w w .op engrou p .org
Prop rietary 64-bit UN IX variant from DEC; now Tru 64
Op en System Interconnect, set of ISO-d efined p rotocols
Ontario Sw ine Im p rovem ent, Inc – w w w .osi.org
Op en Shortest Path First IGP rou ting, RFC 1583,2328
See PDU
Short for PA-RISC, H P‟s CPU chip architectu re
Cisco‟s Port Aggregation Protocol
Practical Extraction and Rep orting (scrip ting) Langu age
Process ID. A u niqu e nu m ber assigned by the OS
A p rogram that issu es ICMP Echo requ ests and rep lies
H P Op enView Problem Diagnosis, an N N M bu ilt -in
Package Data Unit, the OSI Layer 3 d ata encap su lation
Point-to-Point Protocol, RFC 1661, 1990 (Mu ltilink) etc.
Point-to-Point Tu nneling Protocol, RFC 2637
Glossary
PRI
PTP
PTR
RARP
RAMS
RAN CID
rc
Ref
RDMI
RFC
RISC
RIP
RME
RMON
RMON II
RPN
SAA
SAM
SAN S
SE
SEA
sed
SID
SIM
Sim p lex
SMI
SMLT
Solid
SOX
SPAN
SPI
SQL
SMTP
SN MP
SN N P
SSH
SSH 1,2
SSL
syslog
T1/ T3
TAC
TAP
TCP
tail
Prim ary Rate Interface, a typ e of ISDN interface
Acronym for Point To Point
DN S Pointer Record for reverse looku p , RFC 1035
Reverse Ad d ress Resolu tion Protocol, RFC 903, 1931
Op enView Rou te Analytics Managem ent System
Really Aw esom e N ew Cisco confIg Differ
UN IX Rem ote Control files, au to d aem on start and stop
Reference p ages. Accessible from N N M H elp m enu
H P‟s Rem ote DMI access p rotocol
IETF Requ est for Com m ent. Search w eb for RFC 3160
Red u ced Instru ction Set Com p u ting CPU architectu re
Rou ting Inform ation Protocol, an IGP p er RFC 1058
Cisco Resou rce Manager Essentials, p art of CW2K
Rem ote Monitoring SN MP extensions, RFC 1757
Extend s RMON from layer 2 to 3 and m ore, RFC 2021
Reverse Polish N otation, m ath via Postfix N otation
Cisco Service Assu rance Agent; m onitors p erform ance
H P-UX‟s System Ad m instration Manager GUI
SysAd m , Au d it, N etw ork, Secu rity Inst. w w w .sans.org
Refers to H P Op enView N N M Starter Ed ition
Solstice Enterp rise Agent, an SN MP agent, su n.com
Unix stream ed itor com m and
Oracle System Id entifier, u sed to u niqu ely ID a DB
Acroynym for H P System s Insight Manager
synonym ou s w ith H alf Du p lex; one-w ay transm ission
Stru ctu re of Managem ent Inform ation, see RFC 1155
Sp lit Mu lti-Link Tru nking, an aggregate p rotocol for N ortel
Em bed d ed DBMS (w ith N N M). w w w .solid tech.com
Boston Red Sox, 2004 World Series Cham p ions, Yeeha!
Cisco‟s term for p ort sp anning or p ort m irroring
Sm art Plu g-In: ad d -on fu nctionality for N N M, OVO
Stru ctu red Qu ery Langu age, a DBMS access m ethod
Sim p le Mail Transp ort Protocol, RFC 821, 2821, et al
Sim p le N etw ork Mgm t Protocol, RFC 1157, 1905
Sim p le N etw ork Paging Protocol, RFC 1568, 1645
Secu re Shell, generic for SSH 1, SSH 2, Op enSSH , etc
Secu re UN IX logon tool from SCS, w w w .ssh.com
Secu re Sockets Layer, an encryp tion p rotocol, RFC 2246, et al
BSD-based error logging facility, RFC 3164
See DS1/ DS3
Cisco Technical Assistance Center (Su p p ort center)
IXO/ Telocator Alp hanu m eric Protocol, see SN N P
Transm ission Control Protocol, connection -oriented IP
A UN IX com m and that d elivers the last p art of a file
309
Fognet’s Field Guide to OpenView NNM
telnet
Tivoli
Tom cat
Trap
UDP
UCB
UN IX
VAX
Varbind
VLAN
VLSM
VN C
VOIP
VPDN
VRRP
WMI
310
A term inal em u lation p rogram / p rotocol, RFC 854 et.al.
IBM‟s enterp rise m anagem ent p rod u ct su ite
Ap ache‟s Jakarta JAVA servlet
An SN MP u nsolicited notification, p er RFC 1215
User Datagram Protocol, connectionless IP, RFC 768
University of California, Berkeley
A trad em ark of the Op en Grou p , see w w w .u nix.org
A p rop rietary CISC-based OS p rod u ced by DEC
Contraction of Variable Bind ing: SN MP trap attribu tes
Virtu al LAN , logical grou p s of d evices, RFC 3069
Variable Length Su bnet Mask, Internal CIDR,RFC1817
Virtu al N etw ork Com p u ting, w w w .vnc.com
Voice Over IP, Many RFC‟s, e.g. 1889, 2543, 2885, 3261
Virtu al Private Dialu p N etw ork, Cisco, RFC 2764, et al
Virtu al Rou ter Red u nd ancy Protocol, RFC 2338, 3867, 2787
Intel‟s Wired for Managem ent Initiative
Index
$OV_FIELDS, 8
$OV_LOG, 13
$OV_LRF, 5
$OV_REGISTRATION, 6
$OV_SNMP_MIBS, 114
.1.3.6.1.4.1.11.2.17.2.2.0, 109
/etc/ncsd.conf, 36
/etc/netmasks, 118
/etc/networks, 211
1IPathServlet.xml, 253
1NeighborServlet.xml, 251
31 and 32 bit networks, 49
32-bit counters, 159
64-bit counter, 159
64-bit counters, 159
Access Path view, 176
Acknowledged alarm, 110
Action Callback, 164
Action Callback metacharacters, 166
Action Callback Special characters, 166
action callback usage considerations, 165
action callbacks - Java Apps, 222
Action callbacks for a subset of objects, 167
Action callbacks on Windows, 168
add customized symbols, 9
Add Objects Manually, 204
Administrative status, 206
Administrative Status Colors, 110
Advanced Edition, 307
Agent in Distress, 115
aggregate port, 81
Aggregate port scenario, 80
Aggregated port support, 257
alarm, 98
Alarm browser, 110
Alarmpoint, 172
ALLOW_ALL file, 169
Allowable characters in a label, 29
allowOverlap, 276
Amerigo, 183
APA - enable, 74
APA - Filtering by ifType, 90
APA - Filters, 86
APA - firewalls, 91
APA - interface configuration changes, 232
APA - interface table additions, 233
apa - paConfig.xml, 83
APA - polling adjustments, 90
APA - Switching routers, 91
APA - topology events, 92
APA - Troubleshooting, 97
APA aggregated port support, 80
APA Architecture, 71, 72, 74
APA board status support, 81, 82
APA class specifications, 85
APA Connected vs unconnected interface
status, 94
APA Demand Polling, 75
APA important node filters, 93
APA Interface ICMP polling, 84
APA memory footprint, 94
APA performance, 93
APA pickManagementAddesss, 95
APA Poller, 71
APA polling behavior, 95
APA status events, 76, 77
APA Status events, 79
APA status in ovw, 79
app-defaults, 9
Application defaults update procedure, 9
Application Registration Files, 6
Arpanet, 211
ARP-cache, 116
ARPChgNewPhysAddr, 116
ASN.1, 144
audio, 171
Authentication Failure, 112
Authentication Failure events, 119
311
Fognet’s Field Guide to OpenView NNM
AuthFail trap, 120
autozone.ovpl, 226
beep95_lx.sh, 174
BES – increase size, 273
Big Brother, 183
Binary Event Store, 262
BLAT, 172
Bmail, 173
Brass agent, 139
brassd, 142
bridge.noDiscover, 236
Browser cache, 259
cachedb, 262
CAM, 187
Campus Manager, 186
Cannot allocate 128 colors, 21
Cannot convert string, 22
capability fields, 199
Catalyst 2950, 191
CDC, 188
CERT Advisory CA-2002-03, 144
Changing server hostname, 17
Chassis_Temperature, 133
checkDNS.ovpl, 37
checkPollCfg, 78, 95, 238
Chip Sutton, 182
CIA Agent, 152
CIDR, 117
CIM, 162
Cisco, 185
Cisco web links, 196
Cisco chassis event correlation, 133
Cisco discovery configuration, 188
Cisco IOS, 189
cisco IOS commands, 191
Cisco Link Down Trap, 192
Cisco logging levels, 149
Cisco MIBS, 185
Cisco ping MIB, 191
CISCO PIX routers, 115
Cisco Router General Performance
Reports, 270
cisco snmp-server view, 190
Cisco syslog messages, 196
Cisco system stats, 194
Cisco Temperature Probes, 195
Cisco VLANs, 187
Cisco's locIf*BitsSec, 193
312
Cisco-specific event corellation circuits,
196
CiscoSwitchSnmp.user.cfg, 188
CiscoView, 186
CiscoWorks, 186
CiscoWorks web Links, 197
classfulness, 116
Classless Inter-Domain Routing, 118
classless routing, 27
classpath, 222
Cluster virtual IP addresses, 289
clusters, 49
Cold Start, 112
Community String Discovery, 143
Compaq Insight, 162
Compaq Insight Manager, 310
Composer: Enhance, 125; Multi-Source,
125
Composer tracing, 138
Compound Status, 206
Concord eHealth, 184
Concord’s SystemEdge, 152
Connection Editor, 234
connectionEdits, 234
Connector down event correlation, 127
Connector_IntermittentStatus, 132
Container View operations, 246
Container Views, 244
Container Views access control, 247
containers.xml, 245
convertBitmaps.ovpl, 221
cooperative-independent DIM Model, 285
Correlation Composer, 124
Correlation Manager, 124
cpio, 285
Cricket, 184
Critical Path Analysis, 61
cron, 262
Crystal Reports, 266
CS-OV PERL, 182
cu, 173
Customer Views, 308, 309
CW2K, 186
CWSI, 186
Data Collection and Thresholds, 153
Data Collection best practices, 154
Data Collection on High Speed links, 159
data collections - ifSpeed issues, 193
Data Collections on VLANs, 160
Index
Data Warehouse, 262
Database maintenance, 271
Database Queries, 263, 265
Database Size limitations, 273
Databases, 262
Datagram, 161
debug on cisco routers, 180
De-Dup, 131
De-dup correlation, 130
dedup.conf, 127, 132
DeDuplication correlator, 127
Delete From All Submaps, 202
Delete object, 203
Deleted alarm, 110
deleteOVDB.ovpl, 12
deleteSecondary, 276
DHCP, 37
DHCP filters, 277
DIM, 275
DIM Limitations, 275
DIM Overlap Modes, 276
DIM Set-up in a nutshell, 279
Disable Default Reports, 272
Disable Solid, 272
Disable the Data Warehouse, 268, 269
disableNSVandISV.ovpl, 250
DiscoSnmpHelperSchema.cfg, 228
Discovery, 39
Discovery filters, 277
discovery polling algorithm, 47
Discovery polls, 41
disk space, 148
Distributed Internet Discovery, 275
Distributed NNM, 275
DMI, 151
DMI MIF, 151
DMZ, 293
DNS, 25, 30
DNS - Windows, 35
DNS - A record, 36
DNS – Cache Only, 30
DNS - ISC BIND, 32
DNS – Split Horizon, 31
DNS tracing, 37
DNS-BIND, 31
DnsNbtLookupOrder, 34
DnsPriority, 33
doNotDiscoverDuplicates, 50
dot1dBaseNumPorts, 187
dot1dTpFdbTable, 187
DS3, 159
Dupip, 254
dupip.conf, 254
duplicate ifAlias, 28
Duplicate ifAlias, 119
Duplicate_IP_Address, 119
Duplicate_IP_Address events, 118
DuplicateIfAlias, 119
dvclient.jnlp, 243
dvUsersManager.ovpl, 243
dynamic views, 198
Dynamic views, 240
Dynamic Views, 240
Dynamic Views - Troubleshooting, 259
Dynamic views access control, 243, 244
Dynamic Views configuration, 242
Dynamic Views edge connectivity, 260
Dynamic Views features, 240
Dynamic Views registration files, 260
Dynamic Views URLs, 241, 242
dynamic views-based alarm browser, 261
dynamicViews.conf, 242
dynamicViewsUsers.xml, 234, 243
ECS - Permanently Disable, 135
ECS Designer, 126
ECS engine logging, 136
ECS logging, 136
ECS Manager commands, 135
ECS runtime, 123
ECS runtime engine, 126
ECS Tracing, 137
ecscmg.html, 124
ecsevgen, 137
ecsevout, 137
ecsmgr, 127, 135
Edge enPortal and nVision, 183
EGP Neighbor Loss, 112
elm, 170
Email notifications, 169
Email notifications and DNS, 171
Emanate, 152
Emanate extension agent, 141
enableNSVandISV.ovpl, 250
enterprise default, 114
entLastChangeTime, 232
Environment Variables, 3, 4
epoch time, 163
ET, 225
313
Fognet’s Field Guide to OpenView NNM
ET - connections for unsupported devices,
236
ET - End nodes, 235
ET – importing non-SNMP devices, 236
ET and DNS issues, 239
ET and ovw database synchronization, 231
ET autozoning, 226
ET command summary, 237
ET Configuration page password, 234
ET discovery - exclude nodes, 236
ET discovery accuracy, 229
ET discovery performance, 229
ET Manual zoning, 226
ET single/incremental node discovery, 231
ET topology filters, 86
etc/hosts, 25
etc/netmask, 117
etc\networks, 29
Ethereal, 180
Etherpage, 172
etrestart.ovpl, 11
ETsNoSnmpNodes.ovpl, 239
event, 98
event – default event, 114
Event configuration, 99
Event configuration Special Characters,
103
event correlation, 123
Event Correlation, 123
event correlation - summary, 133
event correlation and notifications, 166
event data, 262
Event data export filtering, 273
Event Forwarding, 108
Event logging, 104
event Source External File, 168
Event Variables, 103
eventdb - clear, 274
eventdb - dumping, 107
eventdb - queries, 107
eventdb Increase size, 273
Events - Additional Actions, 111
Events - Dump, 105
events - Launch custom URL’s, 111
Events - NNM-generated, 109
Events OpenView Enterprise, 112
Events to Windows Event, 151
EventsBySeverity.ovpl, 107
evntcmd, 151
314
evntwin, 151
Excel, 268
Exchange, 170
excludeip.conf, 29
Exclusion filter, 278
Expect, 182
exponentially-decayed moving average,
195
Extended Topology, 225, 240
Externally manage objects, 43
Extreme filtering in data exports, 273
Failover filters, 277
Fatal IPMAP, 203
Fault Analysis Tools, 175
fbackup, 285
FDDI, 159
filter expressions, 278
Filtering, 277
filters - external files, 278
filters and wildcards, 278
Find - map operation, 199
firewall, 293
Firewalls, 291
Force or restart an ET discovery, 11
Fragment, 162
Frame, 161
frame relay, 55
frecover, 285
frestore, 285
FRF, 7
fuser, 210
Gambit’s MIMIC, 141
General security, 292
generalConnsEdits, 234
getConnectedPort.sh, 270
gethost, 37
gethostbyaddr, 33, 36
Gethostbyaddr, 25
gethostbyname, 33
global.conf, 270
graphs, 160
HC, 159
hexadecimal IP Address conversion, 191
High Availablity, 291
High Capacity, 159
high speed links, 159
Hiplink, 172
H-Node, 34
homebase, 198
Index
HomeBase, 240
HOST-RESOURCES, 150
Hosts files, 25
hosts.nnm, 233
hot standby, 285
How node objects are labeled, 27
How node objects are named, 25
HP IT Resource Center, 16
HPoid2type, 7
HSRP, 49, 81, 118, 253
HSRP Status, 252, 253
HSRP View, 252, 253
htgroup, 220, 223
I2Trace, 183
IANA, 114
ICMP, 55, 292, 293
ICMP Burst polls, 66
ICMP Redirects, 66
If%util, 154
IF_Down, 129
ifAlias, 28, 29, 52, 65, 79, 97, 119, 193
ifAlias MIB, 190
ifHCInOctets, 159
ifName, 119
ifNumberQueryEnable, 233
IfTypeFilter, 90
ifTypes, 91
ImageMagik, 160, 221
Important Node filter, 128
Important Node filters, 277
InCharge, 183
Inconsistent Subnet Mask event, 116
Integration Packages, 310
Interface index remapping, 193
interface list for all nodes, 274
interface objects, 28
Interface utilization, 153
interface View, 250
Intermittent Status event correlation, 132
Invalid broadcast IP Address, 27
ioscan, 180
IP address table, 146
IP Addresses Wildcards in event source,
168
IP Hostname, 25
ipAddrTable, 293
ipconfig, 35
ipconfig /flushdns, 35
IPMAP, 207, 208, 284
IPMAP Symbol Changes, 205
IPMAP tracing, 211
ipNoLookup.conf, 29
IPv6, 256
IPv6 Status events, 257
IPv6.conf, 256
IPX address, 26
isDiscoContrivedIF, 85
ISDN, 213
Jakarta, 218
JAVA, 24
JAVA console, 259
JAVA heap size, 258
Java Plugin, 24
Java Plug-in Cache, 259
JAVA programming interface, 182
JAVA_HOME, 24
JAVA-based telnet, 222
javaGrapher, 222
Jovw menu registration, 221
jovw registration file, 223
Jovw registration files, 220
JRE, 19
JVM, 222
Kardinia, 184
kermit, 173
keyboard and mouse, 220
Large object database, 214
launcher, 223
Launcher, 221
LD_LIBRARY_PATH, 18
level 2 discovery, 41
Level 3 paths, 175
level 3 topology, 198
Licensing issues, 17
Limiting Discovery, 41
Limiting menu bar access, 214
Link Down, 112
Link Up, 112
linkDown trap - cisco, 192
LLa address, 26
lmhosts, 34
LMhosts, 25
loadhosts, 48, 49
loadOnlyPolledObjectsIntoMemory, 94
Local Registration Files, 5
localtime, 163
log files, 13
logec, 183
315
Fognet’s Field Guide to OpenView NNM
logger, 149
Logging NNM errors to Windows event
log, 13
logindevperms, 171
loopback, 25
Lowest numbered IP Address, 26
MAC Address, 26
MAC addresses, 177
mail, 170
mailx, 170
Maintain/preen web reports, 272
Manage/unmanage objects, 205
Managing Systems using NNM, 148
Managing VLANs, 212
Map - Delete From This Submap, 202
Map - Forcing icons to scale, 210
Map - lining up symbols, 209
Map Application Plane, 200
Map cutting and pasting objects, 198
Map Edit-Add Object, 204
Map Executable, 200
Map Export/Import, 284
Map filters, 277
Map find operation, 199
Map Highlighted, 200
Map IP Map” attributes, 204
map Layout, 200
Map Object Visual Cues, 200
Map Operations, 198
map Overlay, 200
map ownership and permissions, 215
Map Selected, 200
Map selecting multiple objects, 198
Map status progagation rules, 206
map status propagation rules, 206
map Submap persistence, 200
Map Transparent, 200
Map User Plane, 200
map Window Geometry, 200
map.lock, 210
MAPISEND, 170
maxMallocPercent, 160
MC ServiceGuard, 288
menu - limiting access, 214
Mercury Interactive, 1
metacharacters, 166
MgmtAddrInhibited, 96
MgmtAddrMaxSnmpQueries, 96
MIB Application Builder, 6, 260
316
MIB Appls in Dynamic Views, 260
MIB Expression Guru, 156
MIB Expressions, 155
mibExpr.conf, 153, 155
mibtable, 179
Microsoft Excel, 268
migratable IP addresses, 288, 289
migrate70to701ZoneNumber.ovpl, 255
migrateHsrpVirtualIP, 50, 251
Mimic SNMP Simulator, 183
mineETDB.ovpl, 270
Missing charsets in String, 22
MLT.pdf, 258
modem, 173
Motif GUI, 198
mplay32, 172
MRTG, 184
MTU, 161
MultiLink PPP, 50
multilinked routers, 49
Multiple copies of NNM, 275
Multiple reboots event correlation, 133
multiple versions of JAVA, 24
Multi-Router Automatic Protection
Switching, 50
MyHostID.xml, 93
naa.cnf, 141
naaagt, 141
Nagios, 183
Name lookup performance, 33
Name Services, 25
Name services performing poorly, 32
NameSpace.conf, 133
NameSrvQueryCount, 33
NameSrvQueryTimeout, 33
Naming restrictions, 28, 29
NAT, 293
natping, 177
NDBM, 262
ndd, 178
Neighbor View, 251
Netbios, 34
NetBT, 34
netcheck, 179
netfmt, 180
netmon, 39
netmon -c <critical-route-seedfile>, 61
netmon configuration files, 51
netmon discovery polling, 58, 59
Index
netmon -k ConnectorL2Ports, 60
netmon Layer 2 Polling, 59
netmon LRF switch settings, 51, 58
netmon polling statistics, 68
netmon SNMP queries, 52
netmon SNMP-based polls, 60
netmon tracing, 68, 70
Netmon vs. APA, 67
netmon.cmstr, 45, 143
netmon.equivPorts, 202
netmon.MACnoDiscover, 56
netmon.noDiscover, 42, 288
Netmon.noDiscover, 56
netmon.noMACdiscover, 42
netmon.snmpStatus, 61
netmon-loadhosts, 48
netscape, 223
Netscape, 21
NetScout, 186
Net-SNMP, 152, 183
netstat, 178
nettl, 180
nettune, 178
network name, 29
Network named “10”, 211
network objects, 28
network symbols, 211
NIS, 36
NMActiveRoute.Conf, 253
Nmap, 180
nmdemandpoll, 61, 276
nmdemandpoll.ports, 276
NNM 6.2 default data collections, 269
NNM 8i, 295
NNM and multiple NIC cards, 289
NNM APIs, 182
NNM Collection Station, 275
NNM Developer Kit, 182
NNM Filters, 277
NNM Management Station, 275
NNM Patch information, 15
NNM Ports used, 291
NNM Product Documentation, 14
NNM Product Numbers, 311
NNM SPI, 308
NNM UUID, 26
NNM Web Interface, 218
NNM.spec, 261
nnmRptPresenter.exe, 266
No format in trapd.conf, 114
NO_EXTREME_EVENT_FILTERING, 273
Node definition, 307
Node Status, 65
Node status events, 62
Node View, 249, 250
Node_Unknown, 129
Node_Up events, 62
NodeIf event correlation, 129
nodeView, 241
noIpLookup.conf, 36
nolookupdb, 262
non-IP interfaces, 44
Non-IP unconnected port status, 60
nonsnmpnodes.nnm, 236
Nortel MLT and SMLT, 80
NOTIFCATION-TYPE, 114
notification and paging tools, 172
Notifications, 164
NOTIFICATION-TYPE macro, 98, 99
npprobe.log, 176
nscd, 36
nslookup, 35
nsswitch.conf, 35
NTLM, 23
NU0 interfaces, 211
Null interfaces, 211
OAD, 254
Object filters, 277
Object Status, 206
Object-based polling, 58
Octet, 161
ODBC, 268, 272
ODBC Drivers, 263
OID_ALIAS, 192
oid_to_type, 7, 41, 47, 114
Omniback, 282
on-demand submap, 208
oneXmlFileCreator.ovpl, 242
open2, 174
OpenNMS, 183
OpenView Data Protector, 282
OpenView enterprise, 109
OpenView Operations, 148
operational databases, 262
Operational Status Colors, 109
Opnet, 183
Oracle as DW, 267
orphaned objects, 20
317
Fognet’s Field Guide to OpenView NNM
OSPF, 26, 252
OSPF View, 252
Outlook, 170
OV poller plus event correlation, 132
ov.conf, 38, 143, 288, 289
ov.envvars, 4
OV_CONF, 4
OV_EVENT, 98, 109
OV_Node_Down, 130
ovAccessPath.ovpl, 176
ovactiond, 103, 164, 165
ovactiond LRF settings, 167
ovaddobj, 5
ovalarmsrv, 110, 111
ovas, 240, 244
ovas not running, 18, 19
ovautoifmgr, 43, 212
ovbackup.ovpl, 281
ovcapsd, 151
ovcoldelsql, 266
ovcolqsql, 266
ovcoltosql, 157, 266
ovcomposer, 124
ovdbcheck, 266
ovdbdebug, 266
ovdbsetup, 266
ovdelobj, 6
ovdumpevents, 104, 105, 107
ovdupip, 255
ovdvstylereset, 222
ovdwconfig.ovpl, 266, 269, 272
ovdwevent, 266, 271
ovdweventflt, 266, 273
ovdwloader, 266
ovdwquery, 107, 263, 264
ovdwtopo, 266
ovdwtrend, 266, 271
ovdwunloader, 266
Overlapping Address Spaces, 254, 255
OverTime, 184
ovet_apaConfig.ovpl, 39, 74
ovet_bridge, 39
ovet_da* not running, 21
ovet_daHSRPSnmp.log, 254
ovet_demandpoll.ovpl, 75
ovet_dhsnmp.log, 228
ovet_disco, 39
ovet_fixTopology.ovpl, 238
ovet_generateTopoDeltaReport.ovpl, 238
318
ovet_pathengine, 128
ovet_poll, 39
ovet_poll startup time, 230
ovet_reloadTopoDBTbls.ovpl, 238
ovet_topoconndump.ovpl, 235, 270
ovet_topoconnedit.ovpl, 235, 237
ovet_topodump.ovpl, 78, 86, 238
ovet_topoobjcount.ovpl, 238
ovet_topoquery, 235, 238, 253
ovet_toposet.ovpl, 78
ovet_truncatetopotbls_all.ovpl, 239
ovevent, 102
ovexprguru, 156
ovfiltercheck, 277
ovfiltertest, 277
OVHIDESHELL, 169
ovhtpasswd, 220
OVO, 126, 148
ovobjprint, 7, 40
ovotopodump, 307, 308
ovpause, 281
ovrequestd, 262, 268
ovrestore.ovpl, 281
ovresume, 281
ovserver file, 216
OVSHELL, 169
ovsnmp.conf, 262
ovspmd.auth, 216
ovstart, 13, 23
ovstart.allow, 13
ovstatus, 23, 210
ovstop, 211
ovtopodbsnapshot, 285
ovtopodbsnapshot.ovpl, 238
ovtopodump, 20, 40, 203, 274, 277, 280, 301
ovtopofix, 43, 203
ovtrapd, 98, 100, 109, 145
ovw, 198, 200, 215
ovw -fields, 9
ovw – list open maps, 210
ovw map ownership and permissions, 215
ovw map sessions for web GUI, 219
ovw –mapcount, 203
ovw maps – limiting access, 215
ovw symbol status, 207
ovw –verify, 9
OVw*layoutMinSymbol Radius, 210
ovw.auth, 215, 216
ovwchgrp, 215
Index
ovwchmod, 215
ovwchown, 215
ovwdb fields, 23
ovwdb.auth, 216
ovweb.conf, 223
ovwinit failed errors, 203
ovwNTtools.exe, 216
ovwperms, 215
OVwRegDir, 214
OVwRegIntro, 6
Packet, 161
Packet Sniffers, 180
paConfig.xml, 83
pager, 173
PairedTimeWindow, 131
PairWise event correlation, 130, 131
password, 3, 220, 234
passwords, 3
pathping, 179
Paths, 2, 3
Payload, 162
PD, 175
pdcentral.sh, 175
PDU, 161
performance and fault graphs, 156
Performance Insight, 300, 309, 317
Perl, 146, 174
PERL, 47
Perl for action callbacks, 168
persistence filter, 208
Persistence filters, 277
physAddr.conf, 178
PhysAddr_Mismatch, 116
pickSnmpAddrPoll, 45
pickSnmpAddrPolls, 143
Pickup, 170
Ping, 165
ping sweep attacks, 292
Ping Tools, 177
pingPercentRetry, 158
pingResponseTime, 158
pipes, 169
pmd, 137
pmd LRF, 104
pmd Performance, 137
pmdmgr, 137, 180
P-Node, 34
PollerPlus.fs, 133
Polling through firewalls, 276
PollingEngineThreadPoolSize, 94
pop-up menu, 216
port 162, 145, 292
port address mappings, 178
Ports used by MS and CS, 276
Postie, 173
Preferred SNMP address, 45, 46
Preferred IP, 26
preferred IP address, 29
preferred SNMP address, 27
Printing graphs (UNIX), 160
Problem Diagnostics, 175, 307, 309
Product Feature Deltas, 304
propagateStatusForUnconnectedObj, 94
proxy-ARP, 116
PTR record, 36
PTR records, 26
Putty, 182
qpage, 173
RAMS, 309
RDMI, 151
read-write maps – limiting access, 215
recVals.conf, 233
redirects, 169
Re-do discovery (start from scratch), 12
refresh Homebase, 11
RegExp in map find, 199
Regular expressions in find, 199
regverify, 6, 9, 217
reject all traps, 121
release history, 303
Remove a stubborn object, 203
REMOVED objects, 204
repackDb.sh, 271
Repeated event correlation, 130
Reporter URLs, 266
Reporting, 262
Reporting tools, 270
reportPresenter, 271
Reports - delete old, 271
request.properties, 262
request_list, 262
resolv.conf, 35
Resource Manager Essentials, 186
Restore, 281
RFC 1027, 116
RFC 1213, 213
RFC 1215, 98
RFC 1253, 252
319
Fognet’s Field Guide to OpenView NNM
RFC 1493, 213
RFC 1518, 118
RFC 1573, 192
RFC 1850, 252
rfc 2233, 159
RFC 2233, 192
RFC 2578, 98
rfc 2790, 150
RFC 950, 118
RFC1034, 29
RFC1157, 144
RFC1757, 162
RFC2021, 162
ringBell, 165
ringBell.ovpl, 171
RME, 186
RMON, 162
rnetstat, 178
Route Analytics, 309
Router/Switch Health, 129
routing switches, 213, 237
RPN, 156
RRDtool, 183
rrset-order, 31
Scheduled polls, 128
SEC – Simple Event Correlator, 183
seed file, 45
Seeding Discovery, 44, 45
sendmail, 170
sendMsg, 164
sendMsg.ovpl, 100
sendpage, 173
server.xml, 242
ServiceProvider, 34
setstatus.ovpl, 209
setupExtTopo.ovpl, 39, 225
setupSyslog.ovpl, 149
Smart path, 176
Smart Plug-ins, 308
SMON, 162
SMS, 145
SNMP, 112
SNMP – multiple agents, 140
SNMP address - controlling, 143
SNMP agent, 9
SNMP community string, 90
SNMP community strings, 45, 140
SNMP configuration database, 262
SNMP cut views, 190
320
SNMP Data Collection Terminology, 161
SNMP discovery/polling -stop, 44
snmp dot1dTpFdbTable, 229
SNMP Extensible agents, 152
SNMP ExtensionAgents, 141
SNMP HOST RESOURCES MIB, 150
SNMP manager commands, 145
SNMP Manger and Agent functions, 139
SNMP master agent, 152
SNMP Master Agent, 141
SNMP MIB, 40
SNMP OID, 114
SNMP PDU, 144
SNMP Research, 139, 309
SNMP Security Pack, 139, 309
SNMP sysName, 26
SNMP trap, 98
SNMP trap ports, 145
SNMP trap storms, 101
SNMP traps - configuring, 98
SNMP traps - eliminating, 121
SNMP traps - filter out, 100
SNMP traps - generating, 102
SNMP vendor ID, 114
SNMP versions, 140
SNMP versions and history, 139
SNMP vulnerabilities, 144
SNMP Web Resources, 147
snmpColDump, 163, 270
snmpCollect, 153
snmpCollect commands, 157
snmpCollect statistical thresholds, 157
snmpColStats.txt, 157
snmpnolookupconf, 29, 36
snmpnotify, 100, 102
snmpset, 26, 120
snmptrap, 102
SNMPv2c, 112
SNMPv2U, 140
SNMPv3, 141, 142
SNMPv3 and VLANS, 143
snmpviewer, 221
snoop, 180
software loopback address, 26
softwareLoopback, 25
SolarWinds, 183
Solid DB, 262
Solid DB schema, 264
solid.ini, 263, 265
Index
Solidtech, 262
Solistice Enterprise Agents, 152
Sound notifications, 171
SPAN, 160
Spatch, 173
spinning in ifTable, 115
SSH, 217
Stack tracing via pmdmgr, 137
Standard Edition, 307
statTimeRanges.conf, 157
status bridge - disable, 95
status colors, 109
Status event text, 66
Status event variable bindings, 65
Status Polling via netmon, 55
statusAnalyzerThreadPoolSize, 94
submap properties, 200
Sun Cluster, 288
Support Matrix, 304
Support Package Numbers, 313
Swatch, 182
Switching routers, 213, 237
symbol and object delete operations, 202
Symbol Status, 206
Symbol status influences, 207
Symbols superimposed, 212
synchorizing, 202
Syslog integration facility, 148
sysObjectId, 40
System Insight, 162
system performance, 148
system resources, 148
SystemEdge, 152
Systems Insight, 310
Tap Modem Phone numbers, 173
Tavve, 108
Tcpdump, 180
Telalert, 172
Terminal Server, 220
testnonSnmpFile.ovpl, 237
Third Party Tools, 182
time - Epoch, 106
Tivoli NetView, 108
Tomcat, 218
TopN report, 270
topodbpoke, 47
TopoFilters.xml, 86, 230
TopoInventory.ovpl, 270
topology, 262
topology filters, 286
Topology filters, 277
traceroute, 175, 179
tracert.exe, 179
transient submap, 208
Transient, persistent and on-demand
submaps, 208
trap, 98
trap destinations, 142
Trap OIDs - Weird, 191
trap source, 114
TrapBlaster, 183
trapd.conf, 99
trapd.log, 104
TRAP-TYPE macro, 98
trend, 262
Troubleshooting Windows Paths, 10
trunked or meshed connections, 202
trustedCmds.conf, 164
UDP, 144
ufsdump, 285
ufsrestore, 285
ULIMIT, 160
Unable to load any useable fontset, 22
UnconnectedEthIFInConnectedNode, 94
undesirable traps, 121
UNIX clusters, 288
unmanage objects via URLs, 258
unmanaged, 20, 201
unmanageSecondary, 276
Unrecognized ASN.1 type events, 121
useIcmpIfSnmpNoSuchObj, 95
useIfAlias=false, 119
userAuthentication.xml, 243
Using monochrome images, 21
uucp, 173
Variable Bindings, 103
variable-length subnet masks, 117
vendor, 8
vendor ID, 115
Veritas Cluster 2.0, 288
versions, 295
Virtual directories, 17
VivIt, 2
VLAN, 188, 212
VLSM, 117
VNC, 183, 219
VPNs, 212
VRRP, 81, 118
321
Fognet’s Field Guide to OpenView NNM
VRRP View, 251
WanIf, 90
Warm Start, 112
web browser – setting preferred, 222
Web GUI, 218
Web GUI Access Control, 223, 224
Web GUI login password, 220
Web GUI menu and toolbar - adding, 220
Web interface URLs, 218
web.xml, 242, 243
webappmon, 221
Window Scripting Host, 169
Windows Events to NNM, 151
Windows tracing and logging, 13
WinDump, 180
WINS, 25, 33, 34
winsnmp, 145
Wireless, 214
WMI via SNMP, 152
wpaagt, 141
writeSelList, 270
WshShell, 169
wsnmp32.dll, 145
322
Xdefaults, 22
xfs, 23
XNmevent, 110
xnmevents, 99, 164
xnmeventsExt.conf, 111
xnmgraph, 154, 161
xnmgraph command examples, 160
xnmloadmib, 99
xnmpolling, 39, 42
xnmsnmpconf, 37, 61, 120, 140
xnmsnmpconf -clearC, 35
xnmsnmpconf –clearCache, 203
xnmsnmpconf -export, 283
xnmtopoconf, 205, 279
xnmtrap, 100, 108, 164
xpr, 160
xrdb, 22
xterm, 22, 171
xvfb, 219
xwd, 160
Zone Discovery, 233
zones, 226