Pages

Tuesday, June 26, 2012

The Munin graphs are not updating any more

You probably used --force-root with one of the munin programs. You really shouldn't have done that. Now fix the ownership of the .rrd files and of the .png and .html files so that the munin user can write to them again.

In munin 1.2.6 and 1.3.4 there will be a "check-munin" command that checks for correct owner of some files and directories. It does not work well when graph-mode is cgi though.

You can also check your configuration, in munin.conf, if you put server's IP, replace it by 127.0.0.1. (example : instead of 88.191.XX.XX, put 127.0.0.1)

Another possibility is that the structure of the generated files changed from 1.3.x to 1.4.x. Munin might be updating but the files you are looking av are no longer being updated. A quick fix is to remove all files from the munin-html directory and run munin-update again.

rm -rf /var/www/munin-html/*
su -s /bin/bash munin
/usr/bin/munin-cron

check the Logs of the node using

tail -f /var/log/munin/munin-update.log


Writing your own Munin Plugin.

= How to write Munin plugins =

[[TOC]]

Writing a Munin plugin is astoundingly simple.  If you know where the data is and know a minimum of scripting in perl/shell/awk/sed or something like that you can do it.

In way of explaining it all we'll write two plugins the simplest way.  Since I'm kind of old school I'll use shell and shell tools for these plugins.  You can do them completely in perl or any other language you like.

== Load average plugin ==

On Unix you can get a hosts load average from the command ''uptime''. However, on Linux the load average is also available from the file called {{{/proc/loadavg}}}.  One less external command will make the plugin faster.

=== Getting a field value ===

{{{
$ cat /proc/loadavg
0.05 0.07 0.14 1/74 30026
}}}

The file consists of numbers separated by only one space.  The Unix ''cut'' command should be good for that:

{{{
$ cut -d' ' -f1  /proc/loadavg
0.05
}}}

As easy as it gets.  One thing wrong here though: Munin calls the plugins every 5 minutes (this may change in the future).  The better value to report is not the first one but the second one.  In the man page for ''uptime'' you'll see this called the 5 minute load average.

Also, munin wants the value in a more structured form:

{{{
# echo -n "load.value "; cut -d' ' -f2  /proc/loadavg
load.value 0.06
}}}

Here the ''load'' is called the field or field name, ''value'' the attribute, and the number is of course the value. MuninNomenclature has a complete overview of Munin specific terminology.

That was the hard part of the plugin.  The rest is just book-keeping.

=== Munin plugin config command ===

For munin to know how to draw a graph of the reported numbers, it calls the plugin script with ''config'' as the only argument.  A minimal output looks like this:

{{{
graph_title Load average
graph_vlabel load
load.label load
}}}

The global attribute ''graph_title'' sets the title of the graph - in large lettering across the top. The other global attribute ''graph_vlabel'' labels the vertical axis of the graph.  In addition ''load.label'' provides a color coded legend for the graphed line provided by ''load.value''.

=== Everything in a script ===

The most trivial plugin is then this script:

{{{

#!/bin/sh

case $1 in
   config)
        cat <<'EOM'
graph_title Load average
graph_vlabel load
load.label load
EOM
        exit 0;;
esac

echo -n "load.value "
cut -d' ' -f2  /proc/loadavg
}}}

Testing it:

{{{
$ ./load
load.value 0.08
$ ./load config
graph_title Load average
graph_vlabel load
load.label load
}}}

Place the plugin in {{{/etc/munin/plugins}}}. To test it for real use {{{munin-run}}}.  This sets up the environment for the plugin exactly like it would be when run from the [wiki:munin-node munin-node] network service.

{{{
# munin-run load
load.value 0.08
# munin-run load config
graph_title Load average
graph_vlabel load
load.label load
}}}

=== Finishing touches ===

There are a couple more things you can add to improve the plugin.  For example Munin supports more explanatory legends,  and the graph should be tweaked.  For instance, one may add these attributes to the ''config'' output:

{{{
graph_args --base 1000 -l 0
graph_scale no
graph_category system
load.warning 10
load.critical 120
graph_info The load average of the machine describes how many processes are in the run-queue (scheduled to run "immediately").
load.info Average load for the five minutes.
}}}

The values of [wiki:graph_args graph_args] are passed to the graphing tool (rrd) to instruct it about how to draw the graphs.  {{{--base}}} is to make it scale the graph with a 1000 base (1000=1k 1000k=1M and so on.  If you give the base as 1024 as you might when measuring bytes then 1024=1k 1024k=1M and so on. Disks are usually measured in units of 1000 due to the industry standard for marketing disks that people have gotten used to). The ''-l 0'' sets the lowest value to 0.  If all readings of a plugin were between 10 and 100 the lowest value on the graph might otherwise be set to 10.  On a graph showing readings in percent you might add {{{--upper-limit 100}}} (of course some percentage readings goes past 100%).

[wiki:graph_scale graph_scale no] makes munin (rrd) not scale the number.  Normally a reading of 1000 would be scaled to ''1k'' and 1000000 to '1M' (according to scales set with {{{--base}}} explained above).

The ''.warning'' and ''.critical'' attributes are used to issue status messages.  In the case of load average they're probably set statically by the plugin author.  A plugin may also examine the system on which it runs to determine good values for these.  The best way is for the plugin author provide defaults, and then code the plugin to get defaults from environment variables such as $warning and $critical.

The values ''[wiki:graph_info graph_info]'' attribute and each of the ''.info'' field-attributes are added as text on the html page under the graphs.  They serve as legends for the graphs shown.  In the case of this plugin there is not much to say - in other cases, when presenting output from more complex systems (much) more explanation is in order.  Imagine writing for a person that knows Unix/networks/operating systems in general but not the specific sub-system the plugin measures in particular.

This results in a graph such as this:

http://users.linpro.no/janl/.munin/localhost.localdomain-load-day.png

The html load page looks like the [http://munin.ping.uio.no/ping.uio.no/bjarne.ping.uio.no-load.html page pointed to here].  There you can see the use of the info attributes as well.

=== Restarting munin-node ===

The [wiki:munin-node munin-node] network service will not discover new plugins in the plugin directory until you restart it.

{{{
# /etc/init.d/munin-node reload
Stopping Munin Node agents:                                [  OK  ]
Starting Munin Node:                                       [  OK  ]
}}}

Now you can check the reading by telnet:

{{{
# telnet localhost 4949
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
# munin node at foo.example.com
fetch load
load.value 0.06
.
quit
}}}

And now Munin will find the plugin as well - and you should have a readable graph within 15-20 minutes.

Here is the real source for the load average plugin for different architectures:

 * [http://munin.projects.linpro.no/file/trunk/node/node.d.linux/load.in Linux load plugin]
 * [http://munin.projects.linpro.no/file/trunk/node/node.d.freebsd/load.in FreeBSD load plugin]
 * [http://munin.projects.linpro.no/file/trunk/node/node.d.sunos/load.in Solaris load plugin]

You will see that they observe some additional niceties that I've not described here.

== Error handling in plugins ==

The unix way of communicating errors in such small programs is to set the exit code.  Unfortunately just setting the exit code (exit 2, exit 3, exit 4) and not issuing a error message is not very friendly.  Therefore, if there is a execution error, such as not being able to open a file that the plugin should be able to access, please both give a error message and exit with a non-zero value.  In shell it goes like this:

{{{
  if [ ! -r /proc/loadavg ] ; then
     echo Cannot read /proc/loadavg >&2
     exit -1
  fi
}}}

In perl:

{{{
  open(LOAD,"</proc/loadavg") or die "Could not open /proc/loadavg for reading: $!\n";
}}}

== Network interface plugin ==

This plugin demonstrates some additional points but it is much the same as the load plugin.  Firstly this plugin will make two curves in one.  This graph will show how many bytes pass over each network interface on the host it is run.  In quite many Unixes you'll find these numbers in the output of {{{/sbin/ifconfig}}}:

{{{
$ /sbin/ifconfig
eth1      Link encap:Ethernet  HWaddr 00:13:CE:63:45:B2
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::213:ceff:fe63:45b2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2610 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3162 errors:0 dropped:0 overruns:0 carrier:42
          collisions:0 txqueuelen:1000
          RX bytes:817182918 (779.3 MiB)  TX bytes:2835962961 (2.6 GiB)
          Interrupt:18 Base address:0xc000 Memory:b0204000-b0204fff
...
}}}

As usual in Linux though you can also find these numbers in a file.  In this case it's {{{/proc/net/dev}}}:

{{{
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo:22763978  191841    0    0    0     0          0         0 22763978  191841    0    0    0     0       0          0
  eth0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth1:817283042    3242    0    0    0     0          0         0 2836088627    4372    0    0    0     0      42          0
  sit0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
}}}

This is food for awk.  For each interface we're interested in the interface name, received bytes and transmitted bytes. Awk usually uses whitespace as column separator, but this file uses : as well.  Fortunately we can adjust awk's column separator.

{{{
$ awk -v interface="eth1" -F'[: \t]+' \
   '{ sub(/^ */,""); // Remove leading space
       if ($1 == interface) print "down.value "$2"\nup.value "$10;
    }' /proc/net/dev
down.value 818579628
up.value 2837327179
}}}

But there is one important difference: The load plugin reports a number that can simply be plotted on the Y axis.  These ethernet numbers will just continue to grow into the sky as long as the machine is up.  What we actually want to graph is the increase in the numbers between each sampling measured in bits (or bytes) pr. second.  Munin (rrd) will take the number and divide by the number of seconds between the samples (currently the sample interval is fixed at 5 minutes, or 300 seconds), so the "per second" part is taken care of.  The bytes to bits we'll get into in a second.  This is the appropriate {{{config}}} output for the plugin thus far:

{{{
graph_order down up
graph_title eth1 traffic
graph_args --base 1000
graph_vlabel bits in (-) / out (+) per ${graph_period}
down.label received
down.type COUNTER
}}}

The ''COUNTER'' here says that the value is a counter that keeps increasing rather than a GAUGE which the load reading was. If you put this into a script and add a {{{config}}} section you'll have a working plugin producing two curves in one graph.

But there is more:

{{{
down.graph no
down.cdef down,8,*
up.label bps
up.type COUNTER
up.negative down
up.cdef up,8,*
}}}

Here are two or three points. The [wiki:fieldname.cdef .cdef] thing takes care of multiplying by 8 to get from bytes (or bytes pr. second) which is what the file shows to bits (i.e., bits pr. second) which is the unit most humans use when they think about network speeds.

There is a munin policy that input and output to the same device should be graphed in the same graph, this we already do.  BUT, we want the output above the X axis and input below.  This is done by first disabling graphing of the input (downloaded) value, then using [wiki:fieldname.negative up.negative down].  Instead of just negating the down value this keeps the sign and gives the down graph the same color as the up graph.  The values in the database are kept as normal, he whole thing is magiced by Munin while graphing.

The end result is this kind of graph

http://users.linpro.no/janl/.munin/cappuccino.ping.uio.no-if_eth0-day.png

Given a full set of info attributes (I've broken the first line to make it practical to read:

{{{
graph_info This graph shows the traffic of the eth0 network interface. Please note
 that the traffic is shown in bits per second, not bytes. IMPORTANT: Since the
 data source for this plugin use 32bit counters, this plugin is really unreliable
 and unsuitable for most 100Mb (or faster) interfaces, where bursts are expected
 to exceed 50Mbps. This means that this plugin is usuitable for most production
 environments. To avoid this problem, use the ip_ plugin instead.
up.info Traffic of the eth0 interface. Maximum speed is 1000Mbps
}}}

Then you end up with a [http://munin.ping.uio.no/ping.uio.no/cappuccino.ping.uio.no-if_eth0.html generated page like this]

== DERIVE vs. COUNTER ==

To avoid spikes in the graph when counters are reset (as opposed to wrapping), use {{{${name}.type}}} DERIVE and {{{${name}.min}}} 0. Note that this will cause lost data points when the counter wraps, and should therefore not be used with plugins that are expected to wrap more often than be reset (or sampled). An example of this is the Linux {{{if_}}} plugin on 32bit machines with a busy (100Mbps) network.

The reasons behind this is rooted in the nature of 32 bit two's complement arithmetic and the way such numbers wrap around from huge positive numbers to huge negative numbers when they overflow.  Please refer to these two articles in wikipedia to learn more: [http://en.wikipedia.org/wiki/Binary_arithmetic Binary Arithmetic] and [http://en.wikipedia.org/wiki/Two%27s_complement Two's complement].

To summarize:
 1. Use DERIVE
 1. Use {{{${name}.min}}} to avoid negative spikes

== Graph category ==

If the plugin gives the "graph_category" attribute in its {{{config}}} output, the graph will be grouped together with other graphs of the same category.  Please consult (or update) the [wiki:graph_category_list category list].

== Going on ==

Oh yes, I nearly forgot, there are some restrictions on the characters you can use in field names.  They are documented in ''[wiki:notes_on_datasource_names Notes on Field names]''.

The [wiki:Documentation#Plugins plugin documentation] should have all the information you need.  I suggest the next thing you read about plugins is [wiki:plugin-bcp Best Practices] which should tell you all you need to know to get nice graphs in as few tries as possible.  If your plugin does not work like you think it should try [wiki:Debugging_Munin_plugins Debugging Munin Plugins].  To see how get the plugin autoconfigured on install and such take a look at [wiki:ConcisePlugins].

NRPE plug in resulting wrong output.

Hi I am facing a weird issue.

I have installed NAGIOS and using NRPE plugin for remote host monitoring.

I have written a small script to check log files on remote host , the script is working fine from remote host. but from Nagios server ( NRPE plugin the out put is wrong )



Output from Remote host


Server1:/usr/local/nagios/libexec> ./check_batch_status -I air
OK: Batch is successfully submitted for hour 01

NRPE.cfg entry on Remote host


Server1:/usr/local/nagios/libexec> cat ../etc/nrpe.cfg | grep air
command[check_batch_status_air]=/usr/local/nagios/libexec/check_batch_status -I air

Output from Nagios server via NRPE


-bash-3.1$ ./check_nrpe -H Server1-c check_batch_status_air
 CRITICAL:Batch missing for hour 01


Helpdesk Tool

Bugzilla

Bugzilla is a Web-based general-purpose bugtracker tool originally developed and used by the Mozilla project. Released as open source software by Netscape Communications in 1998, Bugzilla has been adopted by a variety of organizations for use as a defect tracker for both free software and proprietary products.

Bugzilla features:
•   Optimized database structure for increased performance and scalability
•   Excellent security to protect confidentiality
•   Advanced query tool that can remember your searches
•   Integrated email capabilities
•   Editable user profiles and comprehensive email preference
•   Comprehensive permissions system
•   Proven under fire as Mozilla's bug tracking system

HP OpenView

HP OpenView Service Desk gives enterprises and service providers a unique solution to successfully manage service levels. The integrated service desk functionality allows you to manage crucial support and service processes to successfully deal with today’s complex IT management challenges.

HP OpenView Service Desk helps you manage your service level agreements (SLAs) by showing you which IT elements your service is depending on, which customers are receiving the service, and who is managing and supporting the service. Now, with new levels of integration with HP OpenView Internet Services Manager and HP OpenView Service Navigator, HP OpenView Service Desk will allow your organization to manage your services better than ever before.

Features:
•   Demonstrates the relationships between customers, business services, servicelevel agreements and support level objectives;
•   Provides a hierarchical service structure with multi-tiered SLA capabilities;
•   Provides straightforward information in a recognizable format with an intuitiveMicrosoft Outlook look-and-feel user interface;
•   Offers the capability to customize forms, views and fields on the fly, withoutthe need to bring the application server down;
•   Provides out-of-the-box reporting capacities and database views to facilitatethe integration with external reporting tools.

BMC Remedy IT Service Management Suite

The BMC Remedy IT Service Management Suite reduces complexity and makes customer support, change management, and asset management integrated and efficient. With BMC, you can benefit from the following:
•   One suite of integrated applications means quicker implementation, accelerated learning curve, and lower TCO
•   One partner for industry-leading software, services, and training means you will fast-track your ITIL initiatives by up to 50 percent
•   One architecture that includes both the leading CMDB and the leading service desk
•   One clear leader in both CMDB and service desk market share


A complete list of help desk software products is present at

http://www.opensourcehelpdesklist.com/

RIMS and RIMS models

A Growing New Model: ‘Asset Light’ RIMO

As CIOs increasingly work with remote infrastructure management tools for configuration, testing and distri¬bution of software, more and more engineers are being empowered to remotely manage data centers, storage, networks, desktop support, help desk and security (i.e., security and control: certified ISO 27000). As CIOs become comfortable with remote management of in¬frastructure planning, administration, monitoring and problem-solving, a new model—remote infrastructure management outsourcing, or RIMO—is emerging to help buyers gain many of the hoped-for benefits of asset ownership transfer without the sometimes nightmarish costs and risks associated with that older model.
RIMO, also known as the ‘asset light’ model, is an increasingly popular form of selective outsourcing in which a solution provider assumes responsibility for managing specific (or all) IT/network functions. Unlike traditional outsourcing arrangements, however, RIMO does not entail the transfer of an enterprise’s assets or personnel to the solution provider or data center. Instead, the organization contracts for a solution pro¬vider to continuously monitor its systems identify po¬tential problems and remediate and resolve real events from a remote location.
The foremost practitioners of RIMO today are managed service providers, or MSPs, who manage IT services for other companies via the Internet. The MSP business model evolved as the traditional computer value-added reseller (VAR) model continually provided lower profit margins. This changing reality, combined with competi¬tion from direct PC vendors such as Dell and Alienware (the latter now owned by Dell) and unpredictable rev¬enues from reactive, break-fix technical support, led to the new model.
In an RIMO scenario, an MSP commonly provides re¬mote network, desktop and security monitoring, patch management, remote data backup and other technical services, monitoring a client’s IT infrastructure and re¬solving any issues that arise within it. Most MSPs charge either a per-month fee, on a time-and-materials basis, or per deployed device (e.g., per server or network device). RIMO is particularly important for the small-to-me¬dium-sized business (SMB) market, giving businesses a relatively cost-effective way to manage their IT without having to hire on-site staff.

RIMS Benefits and Market Growth

According to Everest Research, the RIMO market grew from $720 million in 2005 to $1.5 billion in 2006, and Everest expects the RIMO market to exceed $8 billion somewhere around 2011. RIMO is on the rise because it tends to result in both predictable recurring revenues for the provider and peace of mind for the client, foster¬ing labor arbitrage and delivering greater flexibility and improved process efficiency.
RIMO is inherently more flexible than previous out¬sourcing alternatives. Assets rarely move from home data centers, and the RIMO model is flexible enough that buyers can move assets to nearshore centers in re¬sponse to M&A activity, business growth, increased IT volume or changing technology roadmaps.
RIMO shifts control away from the owners of desktops, servers, mainframes and data centers, and toward the buyers themselves. That helps spur competition that is based on price, capability and cultural fit rather than proprietary vendor requirements.

RIMS Models

‘Asset light’ does not mean ‘asset-free.’ Instead, it means buyers will have more power to retain ownership of their assets so they can not only relieve themselves of vendor control but also break out functions according to core competencies and overall value to the organiza¬tion, and outsource or ‘in-source’ pieces of the IT infra¬structure accordingly. To do this, CIOs will commonly break their overall IT infrastructure into independent ‘towers’ such as network management, server adminis¬tration, maintenance and desktop support. Towers give IT executives the flexibility to pick and choose functions they are willing to farm out to suppliers to get started with RIMO. For example, outsourcing server monitor¬ing within a data center, or quality assurance after code is written, can be a safe way to start. By beginning with one tower at a time, CIOs can familiarize themselves with RIMO providers across different regions with various technology installed bases, as well as varying IT expen¬ditures, resources, governance, and methods of process engineering.
Three RIMO multi-sourcing/’collaborative sourcing’ models are emerging:
Selective/discreet RIMO, where a set of functions and processes is set aside for remote management. That means, for example, that functions such as help desk or data center may be retained by the customer while their monitoring and management is outsourced. This is the best way for most IT executives to get started with RIMO regardless of company size because it involves the least risk: if the first RIMO arrangement does not work, it does not jeopardize the company’s entire IT infrastructure. It also embodies the real promise of RIMO in that the company can outsource pieces of the IT infrastructure to different providers based on their relative capabilities and offers. Full-service RIMO, where a company outsources services but retains the assets and the physical data center. Decisions around full-service outsourcing depend on business case and strategic drivers such as asset refresh and data center consolidation. This scenario is ideal for those companies with the largest embedded base of both IT investment and in-house experience with regard to the equipment itself, but little capability and expertise in application management. This is a good model for nearly all Tier 1 organizations.
Vendor-managed inventory (a.k.a. business-ready infrastructure or BRI), where assets are largely owned by the RIMO provider. BRI combines certified, on-demand application infrastructures and offshore-based services to create a predictable, pay-per-use, multi-year service model and a platform for SaaS that is ideal for the SMB market.

Configuring Nagios plugin for Mozilla Firefox

Nagios plugin for Mozilla Firefox Provides the status report of your Nagios server in the status bar of Browser.
It is really a cool plug-in for any one who is using Nagios as its RIM tool.

To configure it; first you need to
  • install the nagios plugin from https://addons.mozilla.org/en-US/firefox/addon/3607 (https://addons.mozilla.org/en-US/firefox/addon/3607)
  • After you install the plugin ,restart the Firefox, then enable this plug in preferences panel. You do this by going to Tools, then to Add-ons.
  • In setting tab of the plug in select add new connection
  • Name the connection whatever you would like to
  • For the “Nagios web interface url” enter the URL of your Nagios web page, provide the authentication so that Plug in can download the status data from nagios server
  • For the “Status Script url” enter the URL of your status script; If you have not moved or renamed the cgi-bin that nagios uses then
    the default path is http://yourdomain.com/nagios/cgi-bin/status.cgi]http://yourdomain.com/nagios/cgi-bin/status.cgi
  • And save these settings.

Enjoy watching your status of servers in your browser window.

Support Ticket Software

  • A Support Ticket Software is a smart solution to support email queries quickly and efficiently without duplication of effort.
  • The advanced system of email ticketing helps a better communication among customers and system users.
  • Having such a tool is designed to first of all organize all of your support workload and secondly to reduce the amount of time you spend answering customers, (This is a big advantage because if your customers aren’t asking questions it means they are getting all their issues resolved so they don’t have to go through that hassle.)
  • This is an efficient package that incorporates a full email history, group access and multiple user log in, inventory tracking, customized ticket browsing facility, list manipulation, follow up and monitoring the ticket status and many other facilities that truly support service management competently.
  • It helps the customer service team to locate the status of their queries each time, immediately whenever they want. It is a logical and organized approach towards problem management and troubleshooting. By raising trouble tickets that are accepted, and sent for resolving to the relevant system operators, the software actually assigns a system operator with the particular task of handling the problem ticket that is uniquely assigned to the same person. This also ensures that while answering the same no one else in the groups has the same ticket simultaneously.
  • You can install this system on your server easily and effectively. Depending on which one you use it will have comprehensive user interface and automatic tagging facility with unique ticket IDs.
  • Instant Response System or IRS helps in real-time conversation among the seekers and providers. With all the advanced features you can get, responding to customer issues becomes convenient and speedy and you can use the knowledgebase to reply to the inquiries put forward through tickets.
  • Ticket Views assists to set filters the incoming ticket lists. Customizing, Sorting, and using additional filters to organize the ticket for better status review makes the support ticket software indisputable. Login sharing feature makes it easy for clients and users to log in simultaneously without remembering passwords and this can be controlled from a single database.
  • Reporting based on advanced filters employed enables generation of reports sorted by department, date, time, user, or client. The system also makes easy to track the Service Level agreement (SLA) status i.e. whether met timely or not. Additionally, workload analysis and task summary enhances the service management features, by tracking the task/ activity of system operators within a given period in a support ticket software.
  • Purchased Software and Freeware
  • Some of these programs available could be open source and you need no pay anything to use those. Only problem with freeware is sometimes these are crippled versions of what they can exactly do. The basic features are more or less common in a freeware and purchased support ticket software. The purchased products gain an edge because of the value-added features that allow greater flexibility and options. 
Source