Keepalived Vs Heartbeat Difference

A nice explanation highlighting the difference between the keepalived and heartbeat

http://www.formilux.org/archives/haproxy/1003/3259.html

Jenkins as a Centralized Cron Server/Dashboard

h3.  Jenkins

{quote}Jenkins is an open source continuous integration tool written in Java.

Jenkins provides continuous integration services for software development. It is a server-based system running in a servlet container such as Apache Tomcat. It supports SCM tools including CVS, Subversion, Git, Mercurial, Perforce, Clearcase and RTC, and can execute Apache Ant and Apache Maven based projects as well as arbitrary shell scripts and Windows batch commands. The primary developer of Jenkins is Kohsuke Kawaguchi.[3] Released under the MIT License, Jenkins is free software.[4]
Builds can be started by various means, including being triggered by commit in a version control system, scheduling via a cron-like mechanism, building when other builds have completed, and by requesting a specific build URL.{quote}


h3. features :

h5. History :

Jenkins can be configured to retain logs of jobs that it has run. So You have full visibility of what's happening with your system.

Centralized management
With Jenkins you can manage all your imortant cron  jobs centrally.You can also export all Jenkins jobs as XML and check it into a repository.

Alert only if the Jobs fails :
If you enable email for Cron , You end up with too much of emails and eventully end up ignoring them. Jenkins on other hand can be configured to alert only when a particular job fails ie. a job exits with return code other than 0.
In addition you can use the awesome Jenkins Text Finder plugin which allows you to specify words or regular expressions to look for in console output. They can be used to mark a "job" unstable. F

h5. Job dependency :

With Jenkins you can trigger one job only after the first one completes successfully. Creating a well known Dependacy structure and rely on it.

h5. Test immediately :

With Jenkins you can click Build Now and job will run immediately. Where in Cron you have wait for the specified schedule to execute.

h5. Quick  setup

Setting up jobs is easy.




h3. Pros/Cons :

* Backup and restoration becomes very handy.
* We can control the access to Jenkins based on LDAP role.
* Can have multi node Jenkins setup for failover.
* We can schedule the cron job's with dependency i.e. start job B once Job A completes
* Each Job have its history preserved , when did it last run , when failed etc.
* The output of  each job is also captured.
* we can have email alert if the job fails.
* We must have ssh trust between Jenkins server and the target hosts.
* Need to research further.

h3. Implementation method

* h5. Jenkins External Monitor - ( Using your Existing Cron )

# Make Sure the `Jenkins External Monitor Job` plugin is installed.
# Add a New Job in Jenkins, with a type 'Monitor an external job' ( Pick a relevent and name and provide the brief description )
# Now on the remote server
# Make sure the Java is installed
# export the JENKINS_HOME variable JENKINS_HOME=http://host1:8080  ( You can add this info in your .bashrc or Crontab as well )
# If the security on Jenkins server is enabled and only authorized users can run a Job, then JENKINS_HOME should be:
             export JENKINS_HOME=http://user_id:password@host1:8080
# Copy below files from Jenkins server to the remote server in /root/jenkins direcorty
{code} jenkins-core-*.jar remoting-*.jar ant-[1-9]*.jar commons-io-[1-9]*.jar  commons-lang-[0-9]*.jar jna-posix-*.jar xstream-*.jar {code}

# Add below 2 lines to your crontab
        {code}
export JENKINS_HOME=http://user_id:password@host1:8080
* * * * * export JENKINS_HOME=$JENKINS_HOME; java -jar /root/jenkins/jenkins-core-*.jar "Job_Name_In_Jenkins" /path_to/script 2>&1 > /dev/null
        {code}

* h5. Jenkins SSH
# Make Sure the `Jenkins SSH` plugin is installed
# Add the remote hosts details in “Manage Jenkins” –> “Configure System” -> “SSH remote hosts” -> “Add“ ( fill in the details)
# Add new Job in Jenkins , with a type “Build a free-style software project”  ( Pick a relevent and name and provide the brief description )
# Now add in the details for this Job
# Source Code Management ->None
# Build Triggers -> Build periodically -> Schedule -> put a cron schedule
# Build Environment -> Execute shell script on remote host using ssh -> Select the Site added already in above step.
# provide the Pre build script and Post build script , rest all configurations are up to your own requirement.


----

h3. Another proposed solution: Tron

{quote}Tron is a centralized system for managing periodic batch processes and services across a cluster{quote}

h4. The good...

* Manage dependencies between batch process
* A centralized configuration for batches which run on many machines
* Start and restart services running on many machines
* Monitor and report on batch process and services

h4. Tron Architecture

* Tron is a python daemon that executes Jobs or Services across multiple Nodes.
* It is powered by SSH. Meaning Tron just needs to be installed in one centralized location and configured to remotely execute commands.
* Tron manages executing Jobs based on predefined schedules. It resolves dependencies between other jobs in a work flow. It maintains the history of job runs for later reporting. It has a flexible system for monitoring job failures.
* State and job output (stdout/stderr) are stored on disk to avoid loss due to machine failure.
* Tron communicates with command line tools, web interface, or whatever else you can dream up via a REST API.

!http://yelp.typepad.com/.a/6a00d83452b44469e20134871f25c7970c-pi|border=1!

h4. [Tron homepage|https://github.com/Yelp/Tron/wiki]
----

How to install zoner in Centos6

Zoner  is DNS zone management web application built in Python.

{code}

yum install python-setuptools.noarch

easy_install TurboGears

Download the python-tgexpandingformwidget-0.1.3-5.el5ipa.src.rpm in /opt

wget http://ftp.pbone.net/mirror/ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/RHEIPA/SRPMS/python-tgexpandingformwidget-0.1.3-5.el5ipa.src.rpm


rpm -ivh /opt/python-tgexpandingformwidget-0.1.3-5.el5ipa.src.rpm

cd /root/rpmbuild/SOURCES

tar -xvzf TGExpandingFormWidget-0.1.3.tar.gz

python  setup.py install

Download the TGBooleanFormWidget-0.1.egg in /opt

wget http://cleaver.cnx.rice.edu/eggs_directory/TGBooleanFormWidget/TGBooleanFormWidget-0.1.egg#

 easy_install TGBooleanFormWidget-0.1.egg

cd /opt

wget http://peak.telecommunity.com/dist/ez_setup.py

python ez_setup.py -Zf http://peak.telecommunity.com/snapshots/ RuleDispatch

***If you face any error for python.h file install python-devel.x86_64 ( yum install python-devel.x86_64 )

 easy_install easyzone

easy_install dnspython

Finally install zoner -- easy_install zoner

***  python-formencode is dependency so install it ( yum install  python-formencode)



Now configure it --

cd /usr/lib/python2.6/site-packages/zoner-1.4.1-py2.6.egg

cp config/sample-prod.cfg zoner.cfg

Now edit the zoner.cfg and change the parameters

yum install MySQL-python.x86_64


 tg-admin --config=zoner.cfg sql create

zoner_users -c zoner.cfg add

Finally, start the Zoner application:  $ zoner zoner.cfg

Point your browser at http://localhost:8080/ (or the appropriate host/port as per your configuration) and you should be able to login.

Want to run the service other then 8080 port add the below lines in the zoner.cfg 

server.socket_host = "0.0.0.0"
server.socket_port = 9090

Installing and Configuring the DNS slave Server.

h3. Installing and Configuring the DNS slave Server.

We are using the chrooted environment for Bind. so that the base configuration path would be {code}/var/named/chroot/var/named{code}

For non glam dns we will build the caching name server.


h5. Install the required packages

{code}
# yum update -y
# yum install bind bind-utils -y
# yum install bind-chroot -y
#yum install caching-nameserver.x86_64


You should have following packages installed,

bind-chroot-9.3.6-20.P1.el5_8.5
bind-9.3.6-20.P1.el5_8.5
ypbind-1.19-12.el5_6.1
bind-utils-9.3.6-20.P1.el5_8.5
bind-libs-9.3.6-20.P1.el5_8.5
caching-nameserver-9.3.6-20.P1.el5_8.5


{code}

h5. Next step would be to create the named.conf file. Rather than  creating it from scratch, it would be much easier to copy it from any existing slave server.

So take backup of existing /etc/named.conf and scp the  named.conf file from existing slave server at location {code}/var/named/chroot/etc/named.conf{code}

And then create the softlink

{code}
# ls -l /etc/named.conf
lrwxrwxrwx 1 root root 32 Oct 17 00:50 /etc/named.conf -> /var/named/chroot/etc/named.conf

Verify the named configuration, using below command,

# named-checkconf ; echo $?
0

{code}


h5. Master server setup

On master you have to perform two major changes,

* Edit the named.conf and add the IP range of the new slave server in the acl "my_networks"


* Add the NS record of new slave server in every zone file, e.g.

{code}

;  This is a list of all of the named servers for this domain.   The first
;  is the primary (us) and the rest are our various secondaries
;
;  WARNING: cannot list as a nameserver any machine that forwards to mac

@                       IN      NS              tiber.tipsntraps.com
@                       IN      NS              slavetiber.tipsntraps.com


{code}

* Restart the named on both master and slave.

make sure you perform the configtest before restarting the named.


h5. how to perform the confitest

{code}


# named-checkconf ; echo $?
0

# /etc/init.d/named configtest
zone tipsntraps.com/IN: loaded serial 2013121908
{code}


h3. Make sure all the zone files are transferred in the newly configured slave properly, the zone defination files would be located at,

{code}/var/named/chroot/var/named/slaves/{code}

h3. Testing

Change the Serial of one of the zone file on master and do rndc reload , confirm that the change is propagated successfully on the new slave.



h3. Monitoring

We have DNS-Zone-CHECK monitoring in place on master server, which checks that all the zone files defined in master are also setup on the slave server. This monitoring also checks the Serial of every domain of slave server against the master server.

To enable this monitoring, add  the line in file {code}

/etc/check_mk/mrpe.cfg

Note- please pass the appropriate slave server name as second argument

### DNS zone check for rsukapp2
DNS-ZONE-CHECK-slavename /etc/nagios/scripts/check_dns_slave.sh  <master>  <slave>


{code}

once you have added this line in mrpe.cfg, take the re-inventory of the master dns server in check_mk and restart it.

DNS replication slow

Recently at work  I added two new DNS slave servers. We are using BIND 9 as a DNS server.

To my surprise I noticed that the the slave replication on these two boxes were extremely slow, infact after changing the SOA of the zone file it  was taking almost hours to replicate that to these slave boxes.

The other slave boxes were working perfectly fine.

I started digging into it.

/etc/named.conf on both the slaves was looking fine , in fact it was copied from the older running slave servers.

From master the telnet to port 53 was working.

There was no symptoms of any error in the log files.



The gotcha :

after much of debugging and goggling I found the gotcha, that none of the zone files on my master had declared these new slave servers as NS. so ... I added those entries chnaged the serial and restarted named on master. And issue resolved.

@                       IN      NS              tiber1.tipsntraps.com.
@                       IN      NS              tiber1.tipsntraps.com.


Conclusion :

After looking around I found that -

When the serial number is changed on the master, it will notify the slave immediately. In other words,notify is enabled by default. 

BUT, the way notify works is, the master looks at the NS records of that particular domain in the zone file, and notifies the servers listed in the NS record, excluding itself.

if your slave server's hostname is not  listed as an NS record in the zone file;  The slave will  contact the master, listed in the slave's config file - masters { X.X.X.X; }; at the defined Refresh interval.

 And that is why it is taking long time to update to the slave initially. 

YUM rollback in Centos 5 and 6

http://kernelmount.com/content/view/147/9/

Preserving the Environment variable for sudo

There is setting in the suodoers file , using which you can preserve the environment variable while using sudo.


Its called env_keep, see how  I have preserved the YUM0 variable for sudo access.

{code}

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
#
Defaults    always_set_home

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"
Defaults    env_keep += "YUM0"

{code}