Abdul Khader's Home

2010-03-18T03:56:00.001-07:00

Linux: A Platform for the Cloud
Wednesday, 17 March 2010 07:48 Jon "maddog" Hall

The goal of this article is to review the history and architecture of Linux as well as its present day developments to understand how Linux has become today's leading platform for cloud computing. We will start with a little history on Unix system development and then move to the Linux system itself.

Starting Small!

The story of Linux as a platform for cloud computing starts in 1969 with the creation of the Unix2 Operating System at AT&T Bell Laboratories. Unix was first developed on mini-computers, which had very small memory address spaces by today's standards. The PDP-11 (one of the main systems used for the early development of Unix) had an address space of 64 thousand bytes of memory for instructions, and (on some models) 64 thousand extra bytes for data. Therefore the kernel of the operating system had to be very small and lean.

Moving from its original architecture of the PDP-7, onto the PDP-11 (and later onto other architectures), the kernel also divided into architectural independent and architectural dependent parts, with most of the kernel migrating from machine language into the “C” language. The advantage of this architectural move was two-fold: to isolate the parts of the kernel that might be affected by vulgarities in the hardware architecture and to remove as much as possible the tediousness of writing in non-portable machine-language code, which typically led to a more stable operating system.

The kernel of Unix provided only a few “services” for the entire system. The kernel scheduled tasks, managed real memory, handled I/O and other very basic functions. The major functionality of the system was created by libraries and utility programs that ran in their own address spaces. Errors in these non-kernel libraries and utilities did not necessarily cause the entire system to fail, making the system significantly more robust than operating systems that did a great deal of functions in the kernel.

As a time-sharing system it had to have a certain amount of security designed into it to keep one user's data and programs separate from another, and separate from the kernel. The kernel was written to run in a protected space. A certain amount of robustness was also necessary, since a “fragile” operating system would not be able to keep running with dozens or hundreds of users and thousands of processes running at the same time.

Early in the life of Unix, client/server computing was facilitated by concepts like pipes and filters in the command line, and client programs that would talk with server programs called “daemons” to do tasks. Three of the more famous daemons were the printer subsystem, the “cron” (which executes various programs automatically at times specified) and the e-mail subsystem. All of these had “client” programs that would interact with the human on the command line. The client program would “schedule” some work to be done by the server and immediately return to the on-line user. The server programs had to be able to accept, queue and handle requests from many users “simultaneously.” This style of programming was encouraged on Unix systems.

With Unix it was easy and common to have multiple processes operating in the “background” while the user was executing programs interactively in the “foreground.” All the user had to do was put an ampersand on the end of their command line, and that command line was executed in the “background.” There was even an early store-and-forward email system called uucp (which stood for “Unix-to-Unix Copy”) that would use a daemon to dial up another system and transfer your data and email over time.

As Unix systems moved to larger and faster hardware, the divisions of the software remained roughly the same, with additional functionality added as often as possible outside the kernel via libraries, and as seldom as possible inside the kernel. Unix systems had a relatively light-weight process creation due to the command executor's (the “shell”) pipe-and-filter based syntax, so through time, the kernel developers experimented with ever lighter-weight start-up processes and thread execution until Unix systems might be running hundreds of users with thousands of processes and tens of thousands of threads. Any poorly designed operating system would not last long in such an environment.

Unix systems were moving onto the networks of the time, Ethernet and the beginnings of the Arpanet. Design was going into remotely accessing systems through commands like rlogin and telnet, later to evolve to commands like ftp and ssh.

Then Project Athena of MIT offered the Unix world both a network-based authentication system (Kerberos) and eventually the X Window System, a client/server based, architecture neutral windowing system, both continuing the network service-based paradigm. In the last years of the 1990s, many Unix vendors started focusing on server systems, building systems scaling dramatically through Symmetrical Multi-Processing (SMP), high availability through system fail-over, process migration and large, journaled filesystems.

At the start of the twenty-first century Unix systems had become a stable, flexible set of operating systems used for web servers, database servers, email servers and other “service-based” applications. The problem remained that closed source commercial Unix systems were typically expensive, both for vendors to produce and for customers to buy. Vendors would spend large amounts of money duplicating each other's work in ways that the customers did not value.

Large amounts of effort were made in gratuitous changes to the many utility programs that came with Unix. Delivered to customers from various vendors, the commands worked a slightly different way. What customers of the day wanted was exactly the same Unix system across all their hardware platforms.
This general background in mind, we look at the modern-day Linux system and see what Linux offers “cloud computing” above and beyond what Unix offered.

Enter Linux

In 1991, the Linux kernel project was started. Leveraging on all of the architectural features of Unix, the levels of Free Software from GNU and other projects, the Linux kernel allowed distributions of Free Software to take advantage of:

• flexibility in the Unix architecture combined to tailor specific packages to the needs of the user
• lower cost of collaborative development, combined with flexible licensing for service-based support
• same code base across a wide variety of standards-based hardware

Linux continued the overall design philosophies of Unix systems, but added:

• functionality outside the kernel if efficiently possible
• network and API based functionality
• programming to standards

while the “openness” of its development and distribution allows for development and deployment of features and bug fixes outside the main development stream.

Years ago there was a request for journaling file systems in Linux, and several groups offered their code. The mainstream developers felt that the “time was not right,” but the openness of the development model allowed various groups to integrate these filesystems outside of the mainstream, giving customers that valued the functionality the chance to do testing and give feedback on the functionality of the filesystems themselves. In a later release many of these filesystems went “mainstream.”

While not everyone suffers the same extent from the effects of any particular bug, some bugs (and especially security patches) cause great disruptions. FOSS software gives the manager the ability to more quickly apply a bug fix or security patch that is affecting their systems. Linux gives back control to the manager of the system, instead of control remaining in the hands of the manager of the software release.

With potentially millions of servers (or virtual servers) you may get great efficiencies from having distributions tailored to your hardware than the individual software manufacturer would provide. When you have a million servers, a one-percent performance improvement might save you ten thousand servers. There is little wonder why companies like Google and Yahoo use Linux as their base of cloud computing.

In the mid 1990s a concept appeared called “Beowulf Supercomputers,” which later became what today people call “High Performance Computing” (HPC). Most of the worlds fastest supercomputers use Linux so concepts such as checkpoint/restart and process migration started to appear. Management systems evolved that could easily configure, start and control the thousands of machines that were inherent in these HPC systems.

The same basic kernel and libraries used on these supercomputers could also be run on the application developer's desktop or notebook, allowing application programmers to develop and do initial testing of super-computing applications on their own desktop and notebook systems before sending them to the supercomputing cluster.

In the late 1990s and early 2000s, virtualization started to occur with products like VMware, and projects like User Mode Linux (UML), Xen, KVM and VirtualBox were developed. The Linux community led the way, and today virtualization in Linux is an accepted fact.

There are also several security models available. Besides the Kerberos system, there is also Security Enhanced Linux (SELinux) and AppArmor. The manager of the cloud system has the choice of which security system they want to use.

It is also easy to “rightsize” the Linux-based system. The more code that is delivered to a system, the more space it takes up, typically the less secure it is (due to exploits) and the less stable it is (with code that is less-used still being available to create execution faults). FOSS allows a system manager (or even the end user) to tailor the kernel, device drivers, libraries and system utilities to just the components necessary to run their applications, not only on the server side of “The Cloud,” but on the thin client side of “The Cloud,” allowing the creation of a thin client that is just a browser and the essential elements necessary to run that browser, reducing the potential of exploits on the client, all without a “per seat” license to worry about.

If a closed source vendor decides to stop supporting functionality or goes out of business, the cloud system provider has no real recourse other than migration. With FOSS the business choice can be made of continuing that service using the source code from the original provider and integrating that code themselves, or perhaps enticing the FOSS community to develop that functionality. This provides an extra level of assurance to the end users against functionality suddenly disappearing.

Linux provides an opportunity for a cloud service provider to have direct input to the development of the operating system. Lots of closed source software providers listen to their customers, but few allow customers to see or join the development (or retirement) process. The open development model allows many people to contribute. Linux supports a wide range of networking protocols, filesystems, and native languages (on a system or user basis). Linux supports RAID, both software RAID and various hardware RAID controllers.

Linux has a very permissive licensing policy with respect to numbers of machines, of processors per machine and users per machine. The licensing cost in each case is “zero.” While vendors of Linux may charge for support services based on various considerations, the software itself is unrestricted. This makes running a data-center easier than accounting for software licenses on a very difficult licensing schedule as required by some closed source companies.

Finally, size does matter, and while Linux kernels and distributions can be tailored to very small sizes and features sets, Linux was able to support 64-bit virtual address spaces in 1995. For over fifteen years Linux libraries, filesystems and applications have been able to take advantage of very large address spaces. Other popular operating systems have had this feature for just a short time, so their libraries and applications may be immature.

Networking

The area that allows “The Cloud” to work is networking. Linux supports a wide range of network protocols. Linux supports not only TCP/IP, but X.25, Appletalk, SMB, ATM, TokenRing and a variety of other protocols, often as both a client and a server. Early uses of Linux were to act as a file and print server system and email gateway for Apple, Windows, Linux and other Unix-based clients.

Network security features such as VPNs and firewalls delivered in the base distributions combined with the robustness and low cost of the operating system and the low cost of commodity-based hardware to make Linux the operating system of choice for ISPs and Web-server farms in the early 2000s.

More Than Just the Base Operating System

“Cloud Computing” is more than just the kernel and the base operating system. Standard tools are needed on the platform to allow you to develop and deploy applications. Languages associated with “The Cloud” (PHP, Perl, Python, Ruby) started out as FOSS projects and for the most part continue to be developed on FOSS systems. Many of the new web applications and frameworks get developed on Linux first, and then ported to other Unix (and even Windows) systems.

Cloud Frameworks

Even with all these features, Linux would not be as useful for Clouds without some of the cloud framework models that are evolving.

Cloud frameworks typically help in-house systems teams set up and manage “private clouds.” Set up to be compatible with public clouds, instances of virtual environments may be transferred back and forth to allow for local development and remote deployment. Companies may also run the applications in-house under “normal” conditions, but utilize “public cloud” resources under times of heavy load.

Cloud frameworks typically support many styles of virtualized environments with several common distributions. While it is beyond the scope of this article to go into each and every framework, these are two of the main frameworks of today:

Eucalyptus (http://www.eucalyptus.com/)
Eucalyptus is a FOSS Cloud architecture that allows private clouds to be implemented in-house and supports the same APIs as “public” cloud-based environments such as Amazon's Web Services. It supports several types of Virtualization, such as Xen, KVM, VMware and others. Eucalyptus is compatible and packaged with multiple distributions of Linux, including Ubuntu, RHEL, OpenSuse, Debian, Fedora and CentOS.

OpenQRM (http://www.openqrm-enterprise.com/)
OpenQRM is another architecture that allows you to create an in-house “cloud” that supports EC2 standards of APIs. It also supports virtualization techniques such as KVM and Xen to allow you to manage physical and virtual machines and deployments. Virtualized images of Ubuntu, Debian and CentOS are supplied for rapid deployment.

Linux Distributions: Heading into the Clouds

At the risk of missing one of the commercial distributions, this article will mention Ubuntu's Cloud program based on Eucalyptus, Red Hat's Enterprise MRG Grid in conjunction with Amazon's EC2 program, and (while not exactly the same as the first two) SuSE Studio for creating virtualized environments to run under Xen.

Conclusion
It is hoped that this article shows how the architecture of Linux, somewhat guided from its Unix past but enhanced by present day techniques and developments, creates a standard, robust, scalable, tailorable, portable, cost-effective environment for cloud computing, an environment that the cloud supplier and even the end-user can not only “enjoy” but participate in and control.

Acknowledgements
I would like to acknowledge the input of some of the members of the Greater New Hampshire Linux User's Group: Bill McGonigle, Ken D'Ambrosio, Tom Buskey, and Brian St. Pierre in adding to my article about why Linux systems make good cloud computing platforms.

2008-11-06T13:23:00.001-08:00

QASIDA BURDA SHARIF

2008-02-01T16:46:00.001-08:00

Qasida Burda - The poem of the scarf

2008-02-01T16:40:00.000-08:00

Madinah Tun Nabi (PBUH)

2008-02-01T16:38:00.000-08:00

Linux Technical Support

2007-08-08T17:23:00.000-07:00

Linux Technical Support

Call now

$20/Per Hour

Call Me Now

2007-07-06T05:16:00.001-07:00

Get jaxtr | Login

Search Engine

2007-05-13T08:12:00.000-07:00

Hyderabad Weather

2006-05-17T00:58:00.000-07:00


Plan your trip Local Radar Detailed Forecast

2005-10-18T00:26:00.000-07:00

My Photo

Highly-affordable High Availability

2005-05-20T07:05:00.000-07:00

If you're a system administrator, you've already had it happen: you've just ordered lunch when your pager goes off. No lunch for you today. Or maybe you're on the other side of the fence: the server is down, and your system administrator can't be found. You miss your deadline because no one's available to fix your critical system.

High-availability (HA) clusters can dramatically cut downtime, and since service failovers are fast and automatic, system administrators get to finish their lunch and users get to finish their work. "Admins" are happy, users are happy, even pointy-haired managers are happy, because minimizing work stoppages saves money.

Although high-availability means different things to different people, here it refers to highly-available clusters. An HA cluster is a set of servers that work together to provide a set of services. In an HA cluster, services don't belong to any one server in the cluster, but to the cluster as a whole. If one server fails, its services are provided quickly and automatically by another server.

While HA systems can't eliminate outages completely, they can make hiccups very, very short. And when they're short enough, they can go unnoticed or will get blamed on something else -- like a "glitch" in the Internet. When working as it should, an HA system is like an illusionist's trick, where the hand is faster than the eye. Indeed, an HA cluster that's properly designed, configured, installed, and managed should add a "9" to your availability, cutting your downtime by 90%. (See the sidebar "The Magic of Nines" to understand how availability is commonly measured.)

The Magic of Nines

The availability of a service is commonly measured by how many "9's of availability" it provides. If a server is up 90 percent of the time, it has one 9 of availability. If it's up 99 percent of the time, it has two 9's of availability, and so on. If you translate these "number of nines" into how much downtime is allowed per year, you get something like this:

No. of NinesAvailability Downtime/Year

1	90.0000%	37 days
2	99.0000%	3.7 days
3	99.9000%	8.8 hours
4	99.9900%	53 minutes
5	99.9990%	5.3 minutes
6	99.9999%	32 seconds

Even if you start with an unreliable operating system, add unreliable software, and put it on flaky hardware, a good HA cluster package can still make things better. You may get up to three nines if you're lucky. But, if you start with enterprise-class hardware with good maintenance features, add a stable Linux kernel, put on rock-solid applications, mix in some good administrative training and procedures, you can look forward to much better results, perhaps five nines or more.

A Real-life HA Server

Let's see how to configure and deploy a real HA server. The sample cluster [based on the author's personal development cluster] provides four HA services: NFS, Samba, DHCP, and Postfix (a mail relay), and is based on two x86 servers, connected as shown in Figure One. The system can be used to simultaneously develop software, write documents, and store email. It can also be used to serve music files. And since it's an HA system, even if one of the servers crashes or is down for maintenance, the music just keeps on playing. HA email and a bulletproof jukebox -- what more could you ask for?

Figure One: HA cluster physical view

Each server pictured in Figure One is an x86 system running SuSE Linux Enterprise Server 8 (SLES8) with one IDE boot/root drive, and one 80 GB drive for /home. SLES8 was chosen because it comes prepackaged with reasonable versions of all the needed software.

The Heartbeat package is used to detect failures and manage cluster resources. The DRBD package (described briefly here and much more extensively in the feature beginning on page 30) keeps the two copies of /home (one on each server) continually in synch. DRBD can be thought of as RAID1 (mirroring) over a LAN. Each machine is connected to a LAN by a 100 megabit connection, and the two machines are interconnected with a dedicated 100 megabit link for DRBD filesystem synchronization and a serial link for sending heartbeats. Each machine has its own UPS for power protection.

This is a minimal configuration for a high-availability server with shared data. For higher-throughput (write-rate) systems with fast disks, the dedicated link should be a gigabit link. If you use gigabit NICs, they only add a small amount to the cost, and the total cost of putting together such a system remains very low. Exactly how low the system price is depends on what kind of server hardware you start with.

Figure Two: HA cluster service view

Another way to view the system is to see how the various components interact within an active server. Figure Two illustrates that view.

Architecting Your HA Configuration

High-availability clustering is designed to protect your system against failures. So, as you design your own HA system, it's important to look for single points of failure (SPOFs) in your design. If there's a single item whose failure causes the whole cluster to fail, that's a SPOF. The cure for most SPOFs is redundancy. In fact, the "three R's of high availability" systems are Redundancy, Redundancy, and Redundancy. If that sounds redundant, then maybe that's appropriate.

As you look at the system architecture for the sample cluster, you'll see redundant servers, redundant uninterruptible power supplies, redundant disks, and so on. These redundancies are what allows HA clustering to work effectively.

This architecture has no internal SPOFs. No matter what fails in the cluster everything can be recovered. Although the loss of the replication link will stop the data from being replicated to the slave disk, it won't cause system failure, so it isn't a SPOF. (Although we've configured a replication cluster here, shared disks are also commonly used. For a discussion of shared disks versus replicated data, see the sidebar "Shared Disk vs. Disk Replication,") Service can even survive destruction of the primary system by fire.

Shared Disk versus Disk Replication

DRBD replicates data between two disks of any kind and provides very inexpensive storage with no single points of failure. However, it also doubles the storage requirements and incurs some occasionally lengthy resynchronization intervals after crashes. It can also slow down disk writes in some applications.

For many higher-end applications, these disadvantages are troublesome. For those applications, people often use shared disk arrangements instead. These can be multi-attach SCSI RAID boxes, dual controller RAID arrangements (like IBM's ServeRAID), shared fiber-channel disks, or high-end storage like IBM's Enterprise Storage Server, or the various high-end EMC solutions. These systems are relatively costly (ranging from $5K USD to millions of dollars). However, they don't suffer from the latency increases or the more frequent, full resynchs.

Of course, only the most expensive of these solutions avoid internal single points of failure.

Under The Covers: How HA Clustering Works

Figure Three: Normal HA configuration

HA clustering software monitors the servers in the cluster -- typically using a heartbeat mechanism that acts a bit like the Linux init system for the cluster as a whole. That is, the heartbeat starts and stops services so they are always running somewhere in the cluster. One of the most popular HA packages, and the one used in the sample cluster, is called Heartbeat.

Heartbeat uses scripts very similar to standard init scripts to start and stop services. Heartbeat manages resources by groups, and a group of resources always runs on the same machine in the cluster. In addition to normal service scripts (like nfsserver and dhcpd), Heartbeat also manages individual IP addresses as resources through the IPaddr resource script. Resource groups are configured in the /etc/ha.d/haresources configuration file, as explained below.

As mentioned earlier, DRBD is a disk replication package that makes sure every block written on the primary disk gets copied to the secondary disk. From DRBD's perspective, it simply mirrors data from one machine to another, and switches which machine is primary on command. From Heartbeat's perspective, DRBD is just another resource (called datadisk) that Heartbeat directs to start or stop (become primary or secondary) as needed.

Figure Four: Failed over HA configuration

For a cluster providing its services through one IP address, you need three semi-public IP addresses: one for each machine for administrative purposes, and one to talk to the services in the resource group. In the sample cluster, the address 10.10. 10.20 is the service address. That is, whenever anyone wants NFS, Samba, or Postfix services, they connect to 10.10.10.20. Heartbeat makes that IP address available on whatever machine is running the resource group.

In the normal configuration, as shown in Figure Three, paul provides the services and owns the homeserver IP address at 10.10.10.20. If paul fails, silas takes over the homeserver virtual IP address and the corresponding services. If clients try and contact homeserver when paul is down, they reach silas. This situation is illustrated in Figure Four. Now that you know how it all works, here's how to build it.

Prepare the Hardware

There are four cluster-specific things to connect: the disks, the crossover NICs, the crossover serial cable, and the UPS control cables.

* First, install the disks according to usual Linux procedures (see September 2001's "Guru Guidance" column, available online at http://www.linux-mag.com/2001-09/guru_01.html), but don't create any filesystems on them.

* Next, install the NICs, and configure both NICs on private addresses on the same subnet in the ranges in the 192. 168.0.0/16 or the 10.0.0.0/8 range.

Acquire a serial cable intended for PC-to-PC communication. Be sure that the cable includes null modems, and includes the CTS and RTS leads.

Connect each computer to its own UPS.

Although these directions are somewhat x86-specific, all the software runs on all Linux platforms, so you're not restricted to a specific form of hardware.

Install the Software

For this cluster, there are several packages to install. You need: heartbeat-1.0.3, heartbeat-pils-1.0.3, heartbeat-stonith-1.0.3, and drbd-0.6.3. Each is available for SLES8 -- just grab the latest versions from SuSE. If you're not running SLES8, you can get the packages from http://linux-ha.org.

Install the packages using rpm or yast2 or your favorite method. Of course, you'll also need to install whatever services you want to support. For the example, that's nfs-utils, samba, dhcp-base, dhcp-server, dhcp-tools, and postfix.

Configure DRBD

DRBD is configured through the file /etc/drbd.conf. The file has some global parameters and some local parameters. (The drbd. conf file for the example system is shown in "Configuring DRBD.") Make sure to set the disk sizes correctly.

Configuring DRBD

Here's the content of /etc/drbd.conf for the sample configuration.

resource drbd0 {
 protocol=C
 fsckcmd=/bin/true

 disk {
    disk-size=80418208
      do-panic
 }
 net {
      sync-rate=8M # bytes/sec
      timeout=60
      connect-int=10
      ping-int=10
 }
 on paul {
      device=/dev/nb0
      disk=/dev/hdc1
      address=192.168.1.1
      port=7789
 }
 on silas {
      device=/dev/nb0
      disk=/dev/hdc1
      address=192.168.1.2
      port=7789
 }
}

To compute your disk size, use blockdev -- getsize and divide the result by 2. If the two sides give different results use the smaller value.

Next, make a filesystem on paul. It's important that you use one of the journaling filesystems for the filesystem type, and for this example, that you make the partitions exactly the same size.

This means you need to choose one of Reiserfs, Ext3, JFS, or XFS. And, because we're using DRBD, it's safer to make the filesystem on the /dev/nb0 device rather than the underlying device.

Here are the commands to run on paul:

# /etc/init.d/drbd start

When prompted to make paul primary, say "Yes." Next, you need to make the filesystem and mount it.

# mkfs -t reiserfs /dev/nb0 datadisk
/dev/nb0 start

Finally, if you're using a gigabit Ethernet connection for synchronization, change the sync-rate parameter, which limits the maximum speed for resynchronizations.

Configure Heartbeat

Heartbeat has three configuration files: ha.cf configures basic cluster information; haresources configures the init-like resource groups; and authkeys configures network authentication. Sample versions of these files can be found in /usr/share/ doc/packages/heartbeat, and are documented in Heartbeat's "Getting Started" document. These three files need to exist on both machines in the cluster.

ha.cf provides Heartbeat with basic configuration information. It configures the nodes in the cluster, how things should be logged, where to send heartbeats, and parameters concerning the heartbeat interval and dead time interval. This is the /etc/ha.d/ha.cf file for our sample cluster:

logfacility local7# syslog facility
keepalive 1# HB interval
warntime 2# late HB
deadtime 10# failover time
nice_failback on#
node paul silas
ping 10.10.10.254# router addr
bcast eth0 eth1# HB bcast intf.
serial /dev/ttyS0# HB serial link
respawn /usr/lib/heartbeat/ipfail
stonith_host paul apcsmart silas /dev/ttyS1
stonith_host silas apcsmart paul /dev/ttyS1

In the example file above, heartbeats are sent across eth0, eth1, and /dev/ttyS0. For our example (and most clusters), this file is identical across all the nodes. And as noted in the earlier pictures, the power supplies are configured as stonith devices, which are discussed in the "STONITH" sidebar.

STONITH

STONITH is an acronym for "Shoot The Other Node In The Head." It's a technique that Heartbeat uses to ensure that a supposedly dead server doesn't interfere with current cluster operation, and more specifically, that it doesn't damage any shared disks.

If you have shared disks, then STONITH is mandatory. Otherwise, some kind of misconfiguration or software bug might cause each server to think the other side is dead. This is called a

split-brain

condition. If they both mount a shared disk simultaneously, then the data on it is destroyed. This is generally thought to be a bad thing.

There are some types of disk sharing arrangements like IBM's ServeRAID where the hardware guarantees that no more than one computer can access the disk at a time, so they don't need STONITH.

If you're using DRBD, the consequences of split-brain are a little less severe, and for some applications you may be able to ignore them. When using DRBD, a split-brain will cause both sides to become primary and modify their copies of the data separately. Unfortunately, when the two systems come to their senses, you will have to throw away the updates on one of the two systems. If you can live with throwing away good updates during the rare split-brain condition, that cluster can get by without STONITH. If you cannot live with this, then you must configure a STONITH device.

To find out what kinds of STONITH devices Heartbeat currently supports, issue this command:

# /usr/sbin/stonith -L

To get the complete list of information on all these devices and how to configure them, issue this command:

# /usr/sbin/stonith -h

Here's the /etc/ha.d/haresources file:

paul 10.10.10.20 datadisk::drbd0 nfslock nfsserver nmb smb dhcpd postfix

This file creates a single resource group, nominally belonging to paul, containing the IP alias 10.10.10.20, the datadisk (DRBD) resource for drbd0, and the NFS, Samba, dhcpd, and Postfix resources. Heartbeat uses the :: notation to separate arguments to the init scripts. (This is the primary difference between Heartbeat scripts and normal system init scripts.)

To clarify where all these scripts are located, IPaddr and datadisk are located in /etc/ha.d/resource.d/. The other scripts are found in /etc/init.d/, the place normal init scripts are found.

Heartbeat is happy to manage most services that come with init scripts, without any extra work. However, the script names must be identical on all servers in the cluster. (Script names tend to differ between distributions, so using a single distribution across all servers tends to make configuration and maintenance easier.)

Here's the /etc/ha.d/authkeys file:

auth 1
1 sha1 RandomPasswordfc970c94efb

authkeys is the simplest of the configuration files. It contains the authentication method (sha1), and a key to use when signing packets. This file must be identical on all servers in the cluster, and may not be readable or writable by any user other than root.

Configure Services

Services cannot be simultaneously controlled by both Heartbeat and init. Next, disable the nfslock, nfsserver, nmb, smb, dhcpd, and postfix services from starting at boot time. Do that by issuing the following command:

# chkconfig --del nfslock nfsserver  nmb smb dhcpd postfix

Also make sure that the /home partition is not already mounted automatically from /etc/fstab. If there's an entry for /home in fstab, remove it. Next, add an entry like this one:

/dev/nb0 /home reiserfs noauto 0 0

If /home is currently mounted, unmount it.

In most applications, it's necessary to have a name to go with the service IP address. If you use /etc/hosts for your network, you'll need to add a line like this to your /etc/hosts file:

10.10.10.20 homeserver # HA services

If you use DNS, update your DNS servers accordingly. Then clients can add a line like this to /etc/fstab:

homeserver:/home /home nfs \ defaults 0 0

For some services, it's necessary to move their state data to the replicated disk. It's also convenient to move as many HA service configuration files to the shared disk as possible. That way, one copy of these configuration files exists, and you can't accidentally forget to update one of the copies on the cluster.

Next, create a directory called /home/HA-config/. This will mirror portions of the /etc/ and /var/ directory structures. Then move the following files and directories to /home/HA-config/etc/: /etc/postfix/, /etc/samba/, /etc/exports, and /etc/dhcpd. conf, and replace them in the real /etc/ directory with symlinks that point to the pathnames on /home/HA-config/.

Next, do the same thing for the following directories in /var: /var/lib/dhcp/, /var/lib/nfs/, /var/lib/samba/, /var/spool/mail/, and /var/spool/postfix/. The idea of this is that when applications use these files, they will get the files off the replicated /home directory instead of the local root disk.

Next, unmount /home like this:

# datadisk /dev/nb0 stop
# /etc/init.d/drbd stop

Services often need to be told what IP address you want them to use. In the case of nfslock service, the /sbin/rpc.statd program needs to be told the address to advertise NFS locks on by adding the -n homeserver option to the invocation of rpc.statd found in /etc/init.d/nfslock. For Samba, add an interfaces option to the [global] section of /etc/samba/smb.cf:

interfaces = 127.0.0.1/8 10.10.10.20/24

Next, tell Postfix to treat requests coming to the service address as requests from local machines by adding this line to /etc/postfix/main.cf:

inet_interfaces = 127.0.0.1, 10.10.10.20

Testing DRBD

No matter how much you think you know about these services, or configuration, DRBD, or Heartbeat, you must test your HA system. The more thoroughly you test it, the more you'll know about how things work, and the more confidence you'll have in the result. An HA system that isn't well tested won't be highly-available. (Good HA testing could be an article in itself.)

When you use DRBD, you're trusting it to replicate data exactly. It is as vital as the disks and the filesystem code. For now, disable DRBD and Heartbeat from automatically starting with the commands:

# chkconfig --set drbd off
# chkconfig --set heartbeat off

Remember to run the commands on both machines. Now, reboot both servers. On silas, issue this command:

# /etc/init.d/drbd start

On paul, enter:

# /etc/init.d/drbd start

You should see that silas's console has now continued. You can verify that DRBD has made paul primary and silas secondary by running this command on paul:

# cat /proc/drbd

You should see something like this...

0: cs:SyncingAll st:Primary/Secondary

... which indicates that everything's been started correctly, and that a full synch is underway. If you are using a 100 megabit link and large disks, this resynchronization takes a while. You can check progress in /proc/drbd.

To test DRBD, follow the instructions in the DRBD article in this issue, or wait for /proc/drbd to indicate that the full synch is done.

Testing Heartbeat

Next, issue an /etc/init.d/heartbeat start on paul. This starts up the Heartbeat service, producing copious messages in /var/log/messages. To verify that everything's working properly, run the following commands:

# mount | grep /home
# ifconfig | grep 10.10.10.20
# /etc/init.d/nfslock status
# /etc/init.d/nfsserver status
# /etc/init.d/nmb status
# /etc/init.d/smb status
# /etc/init.d/dhcpd status
# /etc/init.d/postfix status

/home should be mounted, the IP address should be set to 10.10.10.20, and all of the services should be running.

Next, start Heartbeat on the other node. Heartbeat will start up like the first node, except it should not start the services.

Migrating Services Manually

Next, tell Heartbeat to move the services from paul to silas by logging into paul, and issuing the command:

# /usr/sbin/heartbeat/hb_standby

Heartbeat quickly moves the entire set of services over to silas. The whole process should take about 15 seconds. Next, log into silas, check the logs, and verify that the services are all running. Next, issue the hb_standby command on silas, to move all of the services back to paul. Check the logs on paul to verify the services are running there again.

Simulating Network Failures

Because there is a ping directive in ha.cf, Heartbeat dutifully pings the router from each machine every second. And because ipfail was started in ha.cf, it monitors the results to see which machine has better connectivity.

At this point, your services should be running on paul. To test ipfail, disconnect paul's eth0 connection; the resources should migrate to silas. Restoring connectivity to paul, and removing it from silas should cause the services to move back.

Simulating Crashes

On to braver tests, and testing crashes. If you followed all of the previous test procedures, the services should be running on paul. Issue the following command on both machines to cause Heartbeat and DRBD to start automatically at boot time:

# chkconfig heartbeat 35
# chkconfig drbd 35

Next, press the reset button on silas. After it reboots, it will start a quick synch with paul, which should complete in a few seconds. After it completes, press the reset button on paul. The services should migrate over to silas with about a ten second delay.

Keeping On

Creating, configuring and testing high-availability systems is an interesting and complex activity that's only touched on here. However, as you can see, for the cost of a serial cable, some NICs, the price of a few hard drives, and a little of your time, you can create an effective HA cluster -- if only with two machines. Read the documentation that comes with Heartbeat and DRBD, and join the Linux-HA and DRBD mailing lists to learn even more. The Linux-HA home page is http://linux-ha.org.

Alan Robertson works for the IBM Linux Technology Center where he is the chief cook and bottle washer (project leader) for the Linux-HA project. He can be reached at alanr@unix.sh.

Getting Started with Linux-HA (heartbeat)

2005-04-24T22:52:00.000-07:00

Getting Started with Linux-HA (heartbeat)

Intro

Let me preface this document by saying most of this is _not_ original work. My purpose for writing this document is just trying to contribute in some way to possibly help those who REALLY get things done. The "work" I am contributing is mostly compiling bits and pieces from other HA documents (such as Volker Wiegand's Hardware Installation Guide) into a document that can help novices get started on HA without pestering Alan (like I did!) and to cut down on repeat questions on the mailing list.

Getting Started

The first thing you'll need is two computers. You need not have identical hardware in both machines (or amount of memory, etc.), but if you did, it would make your life that much easier when a component fails.

Now you have to decide on some of your implementation. Your "cluster" is established via a "heartbeat" between the two computers (nodes) generated by the software package of the same name. However, this heartbeat needs one or more media paths (serial via a null modem cable, ethernet via a crossover cable, etc.) between the nodes.

At this point, you're actually ready to begin hardware-wise. Of course, since you're looking into HA, you'll mostly likely want to avoid having only one point of failure. In this case, that would be your null modem cable/serial port or network interface card(NIC)/crossover cable. So, you need to decide whether you wish to add a second serial/null modem connection or a second network interface card (NIC)/crossover connnection to each node. See Appendix A for instructions on how to build a Cat-5 crossover cable. My heartbeat path setup uses one serial port and one extra NIC because I only had one null modem cable, had an extra of NIC on hand and thought it was good to have two medium types for the heartbeats.

Once your hardware is in order, you must install your OS and configure your networking (I used Red Hat). Assuming you have 2 NICs, one should be configured for your "normal" network and the other as a private network between your clustered nodes (via the crossover cable). For an example, we will assume that our cluster will have the following addresses:

Node 1 (linuxha1): 192.168.85.1 (normal 192x net)
10.0.0.1 (private 10x net for heartbeat)
Node 2 (linuxha2): 192.168.85.2 (192x)
10.0.0.2 (10x)
Note: None of these addresses should be your "cluster address" - the address handled by heartbeat and failed over between nodes!

Most *nix distributions this easy during installation, however, if you are having any problems, refer to either the Ethernet HOWTO, or the documentation for your distribution. To check your configuration, type:

ifconfig

This will show your network interfaces and their configuration. You can obtain your network routing information from "netstat -nr".

If it looks good, make sure you can ping between both nodes on all interfaces.

Next, if you're using one, you'll need to test your serial connection. On one node, which will be the receiver, type:
cat

On the other node, type,:
echo hello >/dev/ttyS0

You should see the text on the receiver node. If it works, change their roles and try again. If it doesn't, it may be as simple as having the wrong device file. Volker's HA Hardware Guide and the Serial HOWTO are two good resources for troubleshooting your serial connection.

Installing Heartbeat.

You can now install the heartbeat package. If you're reading this, you already have it, but in any case it's available at:

http://linux-ha.org/download

There are binary RPMs at the website, or you can build heartbeat from source. Grab the tarball (or install the source RPM). Untar it into your favorite source directory. From the top of the source tree, type "./ConfigureMe configure", followed by "make" and "make install". If you have problems installing the RPMs found at the website and want a way to make your own, there may be help in the FAQ.

Configuring Heartbeat

Configuring ha.cf
There are three files you will need to configure before starting up heartbeat. First, is ha.cf. This will be placed in the /etc/ha.d directory that is created after installation. It tells heartbeat what types of media paths to use and how to configure them. The ha.cf in the source directory contains all the various options you can use, I'll go through it line by line...

serial /dev/ttyS0

Use a serial heartbeat - if you don't use a serial heartbeat, you must use another medium, such as a bcast (ethernet) heartbeat. Replace /dev/ttyS0 with the appropriate device file for your required serial heartbeat.

watchdog /dev/watchdog

Optional. The watchdog function provides a way to have a system that is still minimally functioning, but not providing a heartbeat, reboot itself after a minute of being sick. This could help to avoid a scenario where the machine recovers its heartbeat after being pronounced dead. If that happened and a disk mount failed over, you could have two nodes mounting a disk simultaneously. If you wish to use this feature, then in addition to this line, you will need to load the "softdog" kernel module and create the actual device file. To do this, first type "insmod softdog" to load the module. Then, type "grep misc /proc/devices" and note the number it reports (should be 10). Next, type "cat /proc/misc | grep watchdog" and note that number (should be 130). Now you can create the device file with that info typing, "mknod /dev/watchdog c 10 130".

bcast eth1

Specifies to use a broadcast heartbeat over the eth1 interface (replace with eth0, eth2, or whatever you use).

keepalive 2

Sets the time between heartbeats to 2 seconds.

warntime 10

Time in seconds before issuing a "late heartbeat" warning in the logs.

deadtime 30

Node is pronounced dead after 30 seconds.

initdead 120

With some configurations, the network takes some time to start working after a reboot. This is a separate "deadtime" to handle that case. It should be at least twice the normal deadtime.

hopfudge 1

Optional. For ring topologies, number of hops allowed in addition to the number of nodes in the cluster.

baud 19200

Speed at which to run the serial line (bps).

udpport 694

Use port number 694 for bcast or ucast communication. This is the default, and the official IANA registered port number.

auto_failback on

Required. For those familiar with Tru64 Unix, heartbeat acts as if in "favored member" mode. The master listed in the haresources file holds all the resources until a failover, at which time the slave takes over. When auto_failback is set to on once the master comes back online, it will take everything back from the slave. When set to off this option will prevent the master node from re-acquiring cluster resources after a failover. This option is similar to to the obsolete nice_failback option. If you want to upgrade from a cluster which had nice_failback set off, to this or later versions, special considerations apply in order to want to avoid requiring a flash cut. Please see the FAQ for details on how to deal with this situation.

node linuxha1.linux-ha.org

Mandatory. Hostname of machine in cluster as described by `uname -n`.

node linuxha2.linux-ha.org

Mandatory. Hostname of machine in cluster as described by `uname -n`.

respawn userid cmd

Optional: Lists a command to be spawned and monitored. Eg: To spawn ccm daemons the following line has to be added:

respawn hacluster /usr/lib/heartbeat/ccm
Informs heartbeat to spawn the command with the credentials of that of userid (hacluster, in this example) and monitors the health of the process, respawning it if dead. For ipfail, the line would be:
respawn hacluster /usr/lib/heartbeat/ipfail
NOTE: If the process dies with exit code 100, the process is not respawned.

ping ping1.linux-ha.org ping2.linux-ha.org ....

Optional: Specify ping nodes. These nodes are not considered as cluster nodes. They are used to check network connectivity for modules like ipfail.

ping_group name ping1.linux-ha.org ping2.linux-ha.org ....

Optional: Specify a group ping nodes. These are the similar to ping nodes, but if any node in a group is available then the group is considered available. The group name can be any string and is used to uniquely identify the group. Each group must appear on a separate line. Like ping nodes the group is not considered to be a cluster node. They appear to be the same as ping nodes and are used to check network connectivity for modules like ipfail.

Configuring haresources
Once you've got your ha.cf set up, you need to configure haresources. This file specifies the services for the cluster and who the default owner is.

Note: This file must be the same on both nodes!

For our example, we'll assume the high availability services are Apache and Samba. The IP for the cluster is mandatory, and don't configure the cluster IP outside of the haresources file!. The haresources will need one line:

                  linuxha1.linux-ha.org 192.168.85.3 httpd smb

So, this line dictates that on startup, have linuxha1 serve the IP 192.168.85.3 and start apache and samba as well.
On shutdown, heartbeat will first stop smb, then apache, then give up the IP. This assumes that the command "uname -n" spits out "linuxha1.linux-ha.org" - yours may well produce "linuxha1" and if it does, use that instead!

Note: httpd and smb are the name of startup scripts for Apache and Samba, respectively. Heartbeat will look for startup scripts of the same name in the following paths:
/etc/ha.d/resource.d
/etc/rc.d/init.d

These scripts must start services via "scriptname start" and stop them via "scriptname stop".
So you can use any services as long as they conform to the above standard.

Should you need to pass arguments to a custom script, the format would be:

                scriptname::argument

So, if we added a service "maid" which needed the argument "vacuum", our haresources line would modify to the following:

                linuxha1 192.168.85.3 httpd smb maid::vacuum

This brings us to some added flexibility with the service IP address. We are actually using a shorthand notation above. The actual line could have read (we've canned the maid):

                linuxha1 IPaddr::192.168.85.3 httpd smb

Where IPaddr is the name of our service script, taking the argument 192.168.85.3. Sure enough, if you look in the directory /etc/ha.d/resource.d, you will find a script called IPaddr. This script will also allow you to manipulate the netmask, broadcast address and base interface of this IP service. To specify a subnet with 32 addresses, you could define the service as (leaving off the IPaddr because we can!):

                linuxha1 192.168.85.3/27 httpd smb

This sets the IP service address to 192.168.85.3, the netmask to 255.255.255.224 and the broadcast address would default to 192.168.85.31 (which is the highest address on the subnet). The last parameter you can set is the broadcast address. To override the default and set it to 192.168.85.16, your entry would read:

                linuxha1 192.168.85.3/27/192.168.85.16 httpd smb

You may be wondering whether any of the above is necessary for you. It depends. If you've properly established a net route (independent of heartbeat) for the service's IP address, with the correct netmask and broadcast address, then no, it's not necessary for you. However, this case won't fit everybody and that's why the option's there! In addition, you may have more than one possible interface that could be used for the service IP. Read on to see how heartbeat treats this...

Once you straighten out your haresources file, copy ha.cf and haresources to /etc/ha.d and you're ready to start!

Configuring ipfail
The ipfail plugin attempts to provide detection of network failures, and then intelligently react, directing the cluster to failover resources as necessary. In order to accomplish this goal, it uses ping nodes or ping groups which work as "dumb" third parties in the cluster. Provided both HA nodes can communicate with each other, ipfail can reliably detect when one of their network links has become unusable, and compensate.

To configure ipfail, the following steps must be performed.

Select good ping node candidates.
It is essential that good strategic ping nodes be selected. The better your choices, the stronger your HA cluster becomes. Choosing solid network devices like switches and routers is a good idea. Do not choose either of the members of the HA cluster. Nor should you select someone's workstation. It is also important to select ping nodes that reflect the connectivity of your HA nodes. If you wish to monitor the connectivity of two interfaces, it is wise to select a ping node for each interface, that is reachable exclusively from said interface. Consult ipfail-diagram.pdf for a graphical representation of this idea.
Set auto_failback to on or off.
ipfail will only operate if heartbeat has been configured to something other than legacy In ha.cf, set the auto_failback option to "on" or "off" like so:

auto_failback on

auto_failback off

Configure your ha.cf to start ipfail.
Add a line like the following to ha.cf (assuming your compile PREFIX is /usr)

respawn hacluster /usr/lib/heartbeat/ipfail

Add the ping nodes to ha.cf.
The ping nodes can be added to the cluster by using a line like the following:

ping pnode1 pnode2 pnodeN

Simply replace pnode1, pnode2, ... pnodeN with the IP addresses of your ping nodes.

Ensure that the above configuration directives are added to the ha.cf on both members of the cluster, and that they are identical.

NOTE: You will want to check on the availability of the ping nodes prior to using them. If you cannot ping them from both of the HA nodes, they are useless.

Selecting an Interface

One important aspect of configuring the haresources file for a machine which has multiple ethernet interfaces is to know how heartbeat selects which interface will wind up supporting the service addresses that are configured in haresources. After all, no interface was specified in the haresources file.

Heartbeat decides which interface will be used by looking at the routing table. It tries to select the lowest cost route to the IP address to be taken over. In the case of a tie, it chooses the first route found. For most configurations this means the default route will be least preferred.

If you don't specify a netmask for the IP address in the haresources file, the netmask associated with the selected route will be used. Simmilarly, if an interface is not specivied, then the virtual ip address will be added to the interface associated with the selected route. If the broadcast address is omitted then the hightest address in the subnet is used.

Configuring Authkeys

The third file to configure determines your authentication keys. There are three types of authentication methods available: crc, md5, and sha1. "Well, which should I use?", you ask. Since this document is called "Getting Started", we'll keep it simple......

If your heartbeat runs over a secure network, such as the crossover cable in our example, you'll want to use crc. This is the cheapest method from a resources perspective. If the network is insecure, but you're either not very paranoid or concerned about minimizing CPU resources, use md5. Finally, if you want the best authentication without regard for CPU resources, use sha1. It's the hardest to crack.

The format of the file is as follows:
auth
[]

SO, for sha1, a sample /etc/ha.d/authkeys could be:
auth 1
1 sha1 key-for-sha1-any-text-you-want

For md5, you could use the same as the above, but replace "sha1" with "md5".

Finally, for crc, a sample might be:
auth 2
2 crc

Whatever index you put after the keyword auth must be found below in the keys listed in the file. If you put "auth 4", then there must be an "4 signaturetype" line in the list below.

Make sure its permissions are safe, like 600. And "any text you want" is not quite right. There's a limit to the number of characters you can use.
That's it!

Starting and testing heartbeat

From Red Hat, or other distributions which use /etc/init.d startup files, simply type /etc/init.d/heartbeat start on both nodes. I would recommend starting on the system master (in our example linuxha1) first.

If you want heartbeat to run on startup, what to do will differ on your distribution. You may need to place links to the startup script in the appropriate init level directories, but the RPM versions will do this for you. I have heartbeat start at its default sequential priority (75, which means it starts after services 74 and lower and before services with priority 76-99), end at its default sequential priority (05), and only care about the 0(halt), 6(reboot), 3(text-only), 5(X) run levels.

So, if I had to do it by hand, I'd need to type in the following (as root, of course):

cd /etc/rc.d/rc0.d ; ln -s ../init.d/heartbeat K05heartbeat
cd /etc/rc.d/rc3.d ; ln -s ../init.d/heartbeat S75heartbeat
cd /etc/rc.d/rc5.d ; ln -s ../init.d/heartbeat S75heartbeat
cd /etc/rc.d/rc6.d ; ln -s ../init.d/heartbeat K05heartbeat

The last time I ran slackware, there was no /etc/rc.d/init.d directory (may have changed by now) and to do the same thing, I would have placed in /etc/rc.d/rc.local:
/etc/ha.d/heartbeat start
***This assumes you copy the file ha.rc to /etc/ha.d/heartbeat. If you can't find /etc/rc.d/init.d with your distribution and you're unsure of how processes start, you can use the rc.local method. But you're on your own for shutdown, I just don't remember...

Note: If you use the watchdog function, you'll need to load its module at bootup as well. You can put the following command at the bottom of the /etc/rc.d/rc.sysinit file:
/sbin/insmod softdog
For the rc.local method, just put the same line right above where you start heartbeat.

Once you've started heartbeat, take a peek at your log file (default is /var/log/ha-log) before testing it. If all is peachy, the service owner's log (linuxha1 in our example) should look something like this:
heartbeat: 2003/02/10_13:52:22 info: Neither logfile nor logfacility found.
heartbeat: 2003/02/10_13:52:22 info: Logging defaulting to /var/log/ha-log
heartbeat: 2003/02/10_13:52:22 info: **************************
heartbeat: 2003/02/10_13:52:22 info: Configuration validated. Starting heartbeat 0.4.9f
heartbeat: 2003/02/10_13:52:22 info: nice_failback is in effect.
heartbeat: 2003/02/10_13:52:22 info: heartbeat: version 0.4.9f
heartbeat: 2003/02/10_13:52:22 info: Heartbeat generation: 17
heartbeat: 2003/02/10_13:52:22 info: Starting serial heartbeat on tty /dev/ttyS0 (19200 baud)
heartbeat: 2003/02/10_13:52:22 info: UDP Broadcast heartbeat started on port 694 (694) interface eth1
heartbeat: 2003/02/10_13:52:23 info: pid 28140 locked in memory.
heartbeat: 2003/02/10_13:52:23 info: pid 28137 locked in memory.
heartbeat: 2003/02/10_13:52:23 info: pid 28139 locked in memory.
heartbeat: 2003/02/10_13:52:23 notice: Using watchdog device: /dev/watchdog
heartbeat: 2003/02/10_13:52:23 info: pid 28141 locked in memory.
heartbeat: 2003/02/10_13:52:23 info: Local status now set to: 'up'
heartbeat: 2003/02/10_13:52:23 info: pid 28138 locked in memory.
heartbeat: 2003/02/10_13:52:23 info: pid 28134 locked in memory.
heartbeat: 2003/02/10_13:52:25 info: Link linuxha1.linux-ha.org:eth1 up.
heartbeat: 2003/02/10_13:53:23 WARN: node linuxha2.linux-ha.org: is dead
heartbeat: 2003/02/10_13:53:23 info: Dead node linuxha2.linux-ha.org held no resources.
heartbeat: 2003/02/10_13:53:23 info: Resources being acquired from linuxha2.linux-ha.org.
heartbeat: 2003/02/10_13:53:23 info: Local status now set to: 'active'
heartbeat: 2003/02/10_13:53:23 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2003/02/10_13:53:23 info: /usr/lib/heartbeat/mach_down: nice_failback: acquiring foreign resources
heartbeat: 2003/02/10_13:53:23 info: mach_down takeover complete.
heartbeat: 2003/02/10_13:53:23 info: mach_down takeover complete for node linuxha2.linux-ha.org.
heartbeat: 2003/02/10_13:53:23 info: Acquiring resource group: linuxha1.linux-ha.org 192.168.85.3 datadisk::drbd0 datadisk::drbd1 mirror
heartbeat: 2003/02/10_13:53:23 info: Running /etc/ha.d/resource.d/IPaddr 192.168.85.3 start
heartbeat: 2003/02/10_13:53:23 info: /sbin/ifconfig eth0:0 192.168.85.3 netmask 255.255.255.0 broadcast 192.168.85.255
heartbeat: 2003/02/10_13:53:23 info: Sending Gratuitous Arp for 192.168.85.3 on eth0:0 [eth0]
heartbeat: 2003/02/10_13:53:23 /usr/lib/heartbeat/send_arp eth0 192.168.85.3 00304823BD48 192.168.85.3 ffffffffffff
heartbeat: 2003/02/10_13:53:24 info: Running /etc/ha.d/resource.d/datadisk drbd0 start
heartbeat: 2003/02/10_13:53:24 info: Running /etc/ha.d/resource.d/datadisk drbd1 start
heartbeat: 2003/02/10_13:53:25 info: Running /etc/ha.d/resource.d/mirror start
heartbeat: 2003/02/10_13:53:25 /usr/lib/heartbeat/send_arp eth0 192.168.85.3 00304823BD48 192.168.85.3 ffffffffffff
heartbeat: 2003/02/10_13:53:26 info: Resource acquisition completed.
heartbeat: 2003/02/10_13:53:28 /usr/lib/heartbeat/send_arp eth0 192.168.85.3 00304823BD48 192.168.85.3 ffffffffffff
heartbeat: 2003/02/10_13:53:30 /usr/lib/heartbeat/send_arp eth0 192.168.85.3 00304823BD48 192.168.85.3 ffffffffffff
heartbeat: 2003/02/10_13:53:32 /usr/lib/heartbeat/send_arp eth0 192.168.85.3 00304823BD48 192.168.85.3 ffffffffffff
heartbeat: 2003/02/10_13:53:33 info: Local Resource acquisition completed. (none)
heartbeat: 2003/02/10_13:53:33 info: local resource transition completed.
heartbeat: 2003/02/10_13:56:30 info: Link linuxha2.linux-ha.org:eth1 up.
heartbeat: 2003/02/10_13:56:30 info: Status update for node linuxha2.linux-ha.org: status up
heartbeat: 2003/02/10_13:56:30 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2003/02/10_13:56:30 info: Status update for node linuxha2.linux-ha.org: status active
heartbeat: 2003/02/10_13:56:30 info: remote resource transition completed.
heartbeat: 2003/02/10_13:56:30 info: Running /etc/ha.d/rc.d/status status
heartbeat: 2003/02/10_13:56:31 info: Link linuxha2.linux-ha.org:/dev/ttyS0 up.
NOTE: Your log may differ depending on when you started heartbeat on linuxha2!!! I started heartbeat on the linuxha2 @13:56:30...

OK, now try to ping your cluster's IP (192.168.85.3 in the example). If this works, ssh to it and verify you're on linuxha1.
Next, make sure your services are tied to the .3 address. Bring up netscape and type in 192.168.85.3 for the URL. For Samba, try to map the drive "\\192.168.85.3\test" assuming you set up a share called "test". See Samba docs to get that going. As an aside, however, you'll want to use the "netbios name" parameter to have your Samba share listed under the cluster name and not the hostname of your cluster member!

NOTE: If you can't bring up the service IP address and you get ha-log entries similar to this:

SIOCSIFADDR: No such device
SIOCSIFFLAGS: No such device
SIOCSIFNETMASK: No such device
SIOCSIFBRDADDR: No such device
SIOCSIFFLAGS: No such device
SIOCADDRT: No such device

It may mean that you need to enable IP aliasing in your kernel build. Check /usr/src/linux/.config for "CONFIG_IP_ALIAS=y" if you don't have it, you'll have the line "CONFIG_IP_ALIAS is not set". Rebuild your kernel with IP aliasing enabled.

If this all works, you've got availability. Now let's see if we have High Availability :-)

Take down linuxha1. Kill power, kill heartbeat, whatever you have the stomach for, but don't just yank both the serial and eth1 heartbeat cables. If you do that, you'll have services running on both nodes and when you re-connect the heartbeat, a bit of chaos....
Now ping the cluster IP. Approximately 5-10 seconds later it should start responding again. Telnet again and verify you're on linuxha2. If it happens but takes more like 30 seconds, something is wrong.

If you get this far, it's probably working, but you should probably check all your heartbeats, too.
First, check your serial heartbeat. Unplug the crossover cable from your eth1 NIC that you're using for your bcast heartbeat. Wait about 10 seconds.
Now, look at /var/log/ha-log on linuxha2 and make sure there's no line like this:
1999/08/16_12:40:58 node linuxha1.linux-ha.org: is dead
If you get that, your serial heartbeat isn't working and your second node is taking over. To avoid any problems, shut down heartbeat on the first node, then test your null modem cable. Run the above serial tests again.

If your log is clean, great. Re-connect the crossover cable. Once that's done, disconnect the serial cable, wait 10 seconds and check the linuxha2 log again.
If it's clean, congrats! If not, you can check /var/log/ha-log and /var/log/ha-debug for more clues.

How to authenticate Linux box with windows NT/ 2000 server

2005-01-19T22:39:00.000-08:00

Please follow the steps as I have mentioned step by step.

1. Using the appropriate Microsoft utilities, add the Desktop
workstation to the domain. For example, for a Windows 2000 domain, run
Active Directory Users and Computers and add the Desktop machine to the
"Computers" OU (or wherever is appropriate for your environment), making
sure to turn on "Allow pre-Windows 2000 computers to use this account".

NOTE : When adding the computer to the active directory please click on
"Allow pre-Windows 2000 computers to use this account".

2. Please copy nsswitch,conf to /etc . Please copy kde, login, samba and
xscreensaver to /etc/pam.d and set the permissions as of the original files.
Please take a backup of all these original files before you overwrite them.

3. Copy the smb.conf to /etc/samba

4. Edit smb.conf and make changes to the following line as suitable for
your environment.

workgroup = KHADER

Note: (Here, I assume the user is using Komba. Else the user can use windows network neighborhood) To check the domain name please start your My Network and then in
that you will see the domain name, please use that name as visible in
the My Network and when you click on that domain the domain controller's
netbios name will also appear, please use that name as the netbios name.
Here I will give my details on the server and how they were visible in
the My Network. I used the details of My Network and everything worked

ON WINDOWS 2000 SERVER(When logged in as Administrator)

Rightclick on My Computer
Click on Properties
Click on Network Identification
You will see the following:

Full Computer Name : win2000.khader.jit.com (This my configuration, you
might see something else)

Domain : khader.jit.com

But when I start My Network I see this domain as KHADER and PDC as
win2000

5. Start the winbind service.

To join the system to the domain, give the following command

smbpasswd -j khader -r win2000 -U Administrator

Here khader will be replaced by your domain name and win2000 will be
replaced by the netbios name of the pdc

Then give the following commands

wbinfo -u
wbinfo -g
wbinfo -t

The last command should give you he following message

Secret is Good

Then give this command

wbinfo -a KHADER+test%test

KHADER is replaced by your domain name and test by the domain user and
the test password accordingly. This should give you the following
message.

plaintext password authentication succeeded

6. Create directory /home/WINDOWSDOMAIN (where WINDOWSDOMAIN is the
NetBIOS name of your Windows domain. USE CAPITAL LETTERS.)

7. Then Create /usr/local/bin/phdfwda

#!/bin/sh
#phdfwda

if ! [ -f "$HOME/Desktop/Personal Files" ] ; then
cp -R -u -d /etc/skel/* "$HOME"
fi

Run chmod a+x /usr/local/bin/phdfwda

8. Create /opt/kde2/share/autostart/phdfwda.desktop

[Desktop Entry]
Name=Populate home directory for Windows domain accounts
Exec=phdfwda
Type=Application
X-KDE-StartupNotify=false

Now go to the login prompt (X or text based) and type like the
following.

Login: KHADER+test
Password:

Please use DOMAINNAME+DOAMIN_USER_NAME

It will log you on. That's it. From now on you will be authenticated
from windows server and not the linux box. It will authenticate you from local machine if the domain controller is not available.

Please find below the names and contents of the files to be changed.

nsswitch.conf
============

#
# /etc/nsswitch.conf
#
# An example Name Service Switch config file. This file should be
# sorted with the most-used services at the beginning.
#
# The entry '[NOTFOUND=return]' means that the search for an
# entry should stop if the search in the previous entry turned
# up nothing. Note that if the search failed due to some other reason
# (like no NIS server responding) then the search continues with the
# next entry.
#
# Legal entries are:
#
# nisplus or nis+ Use NIS+ (NIS version 3)
# nis or yp Use NIS (NIS version 2), also called YP
# dns Use DNS (Domain Name Service)
# files Use the local files
# db Use the local database (.db) files
# compat Use NIS on compat mode
# hesiod Use Hesiod for user lookups
# [NOTFOUND=return] Stop searching if not found so far
#

# To use db, put the "db" in front of "files" for entries you want to be
# looked up first in the databases
#
# Example:
#passwd: db files nisplus nis
#shadow: db files nisplus nis
#group: db files nisplus nis

passwd: compat winbind
shadow: compat
group: compat winbind

#hosts: db files nisplus nis dns
hosts: files dns

# Example - obey only what nisplus tells us...
#services: nisplus [NOTFOUND=return] files
#networks: nisplus [NOTFOUND=return] files
#protocols: nisplus [NOTFOUND=return] files
#rpc: nisplus [NOTFOUND=return] files
#ethers: nisplus [NOTFOUND=return] files
#netmasks: nisplus [NOTFOUND=return] files

bootparams: nisplus [NOTFOUND=return] files

ethers: files
netmasks: files
networks: files
protocols: files
rpc: files
services: files

netgroup: files

publickey: nisplus

automount: files
aliases: files nisplus

==================================================================

kde
====

#%PAM-1.0
auth required /lib/security/pam_nologin.so
auth sufficient /lib/security/pam_winbind.so
auth required /lib/security/pam_pwdb.so use_first_pass shadow nullok
account required /lib/security/pam_winbind.so
password required /lib/security/pam_cracklib.so type=user retry=3
password required /lib/security/pam_pwdb.so use_authtok
session required /lib/security/pam_mkhomedir.so skel=/etc/skel umask=0022
session required /lib/security/pam_pwdb.so

=================================================================

login
====

#%PAM-1.0
auth required pam_securetty.so
auth required pam_nologin.so
auth sufficient pam_winbind.so
auth required pam_pwdb.so use_first_pass shadow nullok
auth optional pam_mail.so
account required pam_winbind.so
session required pam_mkhomedir.so skel=/etc/skel umask=0022
session required pam_pwdb.so
session optional pam_lastlog.so
password required pam_pwdb.so

=============================================================

samba
======

#%PAM-1.0
auth sufficient pam_winbind.so
auth required pam_pwdb.so use_first_pass nullok nodelay
account sufficient pam_winbind.so
account required pam_pwdb.so nodelay
session required pam_pwdb.so nodelay
password required pam_pwdb.so shadow md5

=================================================================

smb.conf
========

[global]

domain master = No
local master = No
password server = *
security = domain
template homedir = /home/%D/%U
template shell = /bin/bash
winbind enum groups = yes
winbind enum users = yes
winbind gid = 10000-20000
winbind separator = +
winbind uid = 10000-20000
# workgroup = NT-Domain-Name or Workgroup-Name
workgroup = KHADER
server string = Member of win2000
encrypt passwords = yes

====================================================================

xscreensaver
===========

#%PAM-1.0

# Red Hat says this is right for them, as of 7.3:
auth sufficient pam_winbind.so
auth required pam_pwdb.so use_first_pass shadow nullok
# This is what we were using before:
# auth required pam_pwdb.so shadow nullok

=================================================================

Installing fonts in Linux and Mozilla

2005-01-19T22:30:00.000-08:00

Installing fonts in Linux and Mozilla
==========================

Mozilla fonts suck big time. Especially if you visit a page with Trebuchet MS font like I've recently started using on mahmood.tv. True to my promise of using Linux exclusively over the next couple of weeks, I thought I'd better do something about the font display on my linux laptop.

In order to do that, I needed to copy the fonts I use for Windows XP Pro to Linux, that started the journey, however I must say that there is a wealth of information in this regard on the net, all you have to do is search for it.

Here are the steps I did to get my fonts over to Linux and how I activated them:

1. if you don't have a /usr/share/fonts/TrueType directory, create one
2. make sure that all the font files you're copying have lower-case extensions, so if you have files like
font.TTF, rename that to font.ttf
3. copy the fonts to /usr/share/fonts/TrueType
4. change into that directory: cd /usr/share/fonts/TrueType
5. issue the following commands to build the necessary fonts.scale and fonts.dir:

ttmkfdir -o fonts.scale

mkfontdir

6. this should have now created the fonts.scale and fonts.dir files
7. now go to you /etc/X11 directory so that you can add the new font directory to the search path of your
X server by adding it to the fonts section of XF86Config
8. edit XF86Config
go to the �Files� section and add the FontPath there, mine looks like the following
after editing:

RgbPath "/usr/X11R6/lib/X11/rgb"

FontPath "unix/:7100"

FontPath "/usr/share/fonts/TrueType"

9. Now for the Mozilla part! Go to the directory you installed Mozilla in, mine is located
at /usr/local/mozilla. Change into /usr/local/mozilla/defaults/prefs
10. backup the �unix.js� preference file so that you can get back to it if anything screws
up:
11. cp unix.js unix.js.bak
12. edit the unix.js file and look for the �TrueType� section and change it to look like
the following snippet:

// TrueType

pref("font.FreeType2.enable", true);

pref("font.freetype2.shared-library", "libfreetype.so.6");

// if libfreetype was built without hinting compiled in

// it is best to leave hinting off

pref("font.FreeType2.autohinted", false);

pref("font.FreeType2.unhinted", false);

// below a certian pixel size anti-aliased fonts produce poor results

pref("font.antialias.min", ;

pref("font.embedded_bitmaps.max", 1000000);

pref("font.scale.tt_bitmap.dark_text.min", 0);

pref("font.scale.tt_bitmap.dark_text.gain", "0.7");

// sample prefs for TrueType font dirs

pref("font.directory.truetype.1", "/usr/share/fonts/TrueType");

pref("font.directory.truetype.2", "/usr/X11R6/lib/X11/fonts/TTF");

//pref("font.directory.truetype.3", "/u/sam/tt_font3");

pref("font.FreeType2.printing", true);

Now you'll enjoy a MUCH clearer veiw of websites you visit!

I'm still not very happy as the fonts are not kerned properly and look a bit spaced out on my laptop. I'll
have a look at that next and let you know if I get better display.

For OpenOffice.org, you can simply run the �spadmin� program and add your fonts by pointing to the TrueType directory and �add all�, close the panel and start up OpenOffice and you'll be happy to see that you now have all the fonts to play with in your documents

My Resume

2005-01-14T23:55:00.000-08:00

Abdul Khader
Objective
To work as a System Administrator
Education
1999 [October]
* Completed Caldera Systems Authorized Course: Linux Administration for Networking Professionals from Swarnasoft Solutions Pvt. (Ltd.), [Hyderabad]
1999 [September]
* Microsoft Certified System Engineer
* Windows NT Server 4.0 in the Enterprise
* Windows NT Server 4.0
* Windows NT Workstation 4.0
* Networking Essentials
* TCP/IP
* Internet Information Server 4.0
1999 [August]
* Attended Workshop on Windows 2000 (NT 5.0) conducted by KAROX Technologies Ltd.
* Installation, Configuration, ADS, DCs, OUs, Trees, Forests, RAS, NTFS, TCP/IP, DNS, WINS, IIS5, IE5.
1993 - 1995
* Advance Diploma In Systems Management from NIIT, Hyderabad.
1992 - 1994
* M.A [Eng. Lit.] from Osmania University Hyderabad.
1989 - 1992
* B.A from Osmania University Hyderabad.
Awards received
Gold Medal in B.A
Languages
English and Hindi

Work experience

1998 [October] - 1999 [June]
Worked as an Administrator and taught Windows NT in Microtek Computer Education Center, [Hyderabad].
My responsibilities included planning of network, implementing the network, making system policies and user policies for different users and groups, securing the resources of the network planning the security and implementing the security, to monitor the network and trouble shoot any problems of the users, add users, assign their permissions and design their interfaces, add new clients to the network, planning for the installation and installing Windows NT Server and Workstation, install services like DHCP Server and configure it, Create the scope for DHCP according to the requirements, trouble shoot boot-up problems and printing problems, print job scheduling, replicating files and other administrative activities. I also have a sound knowledge of Windows 95 / 98 / windows Millennium including installation, Microsoft Excel and Microsoft Word 97.

1999 [October] - 2000 [July]
Worked as an Administrator in Swarnasoft solutions Pvt. (Ltd.).[Hyderabad]
My responsibilities included going to client sites, planning of network, implementing the network, making system policies and user policies for different users and groups, securing the resources of the network planning the security and implementing the security, to monitor the network and trouble shoot any problems of the users, add users, assign their permissions and design their interfaces, add new clients to the network (Linux and Windows95/98/NT), and
Sharing data across Linux and Windows clients, planning for the installation and installing Linux server and Workstation, install services like DHCP and DNS Server and configure it, Create the scope for DHCP according to the requirements, install and configure SAMBA, install and configure APACHE web server, install and configure SQUID (Linux native proxy server) trouble shoot boot-up problems and printing problems, print job scheduling and other administrative activities. I also present demonstrations of Linux and give presentations in seminars conducted by the company to promote Linux.

* Projects completed:
1) Project Location:
Advanced Radio Models (A.R.M)
Project Requirements:
Intranet with Proxy, NFS, NIS, Star Office, SAMBA

2) Project Location:
Andrapradesh Technology Services
Organization which is directly responsible for
Imparting IT education to all of the state govt.
Officials.
Project Requirements:
Give technical demonstration(all about Linux
including the internals) and install Linux, Apache,
Samba, Squid, NIS, DHCP, Star office etc.

3) Project Location:
Swarnasoft Solutions Pvt. Ltd.
Swarnasoft which is the only authorized teaching
center and authorized technical support organization
for Caldera Systems Inc. USA [One of the major
distributors of Linux in the World], in South Asia.
Project Requirements:
Make a purely functional Clustering system on Linux.
Clustering requires more than one system to process
a given job simultaneously, dividing the given job into
chunks and allotting each node a chunk of the job to
be processed. This is also better known as parallel
processing.
* This project was in its final phase.
4) Project Location:
Andhra Sugars Pvt. Ltd., Tanuku, East Godavari,
A.P, India.
Project Requirements:
Installing Linux (SuSe, Corel, Redhat, Mandrake and
Caldera),
Configuring Samba, Apache, sendmail, Squid Proxy
Server, Webmin, NFS, DNS, DHCP.
Taught all the above mentioned to the IT Staff of
Andhra Sugers Pvt. Ltd.

5) Project Location:
Zen Computers Pvt. Ltd., A.P, India.
Project Requirements:
Planning the network and implementing the network.
Installing Windows NT and Windows 98. Installing NT
server 4.0 on two PDC's (Multiple Domain Network)
and two BDC's and installing and configuring Microsoft
Proxy Server and Win Proxy, installing and configuring
DHCP, DNS, RAS, Multilinking, Securing the network
and setting up network monitoring tools. Installing
Windows 98 on 25 systems.

2000 [July] - 2000 [November]
Worked as an Administrator and teaching Windows NT, 95, 98 and Linux in Expert Software Services Pvt. (Ltd.).[India].
1) Project Location:
Expert Software Services Pvt. Ltd., A.P, India.
Project Requirements:
Planning the network and implementing the network.
Installing Windows NT and Windows 98 and Linux.
Installing Caldera Open Linux e-Server 2.3, Windows
NT 4.0 and Windows 95 and 98.
installing and configuring Samba, Webmin, Squid,
Quota Allocation, Apache Web Server, Sendmail,
NFS, DHCP, Installing and configuring WebRam,
Multilinking. Securing the network and setting up
network monitoring tools.

2) Project Location:
Deccan Computers Pvt. Ltd., A.P, India.
Project Requirements:
Planning the network and implementing the network.
Installing Windows NT and Windows 98 and Linux.
Installing NT Server 4.0 on one PDC and oneBDC and
installing and configuring Comsocks Proxy Server and
configuring DHCP, Multilinking, Securing the network
and setting up network monitoring tools.

3) Project Location:
Cyber-dyne Systems Pvt. Ltd., A.P, India.
Project Requirements:
To make a complete Linux based network.
Planning the network and implementing the network.
Installed Caldera Open Linux e-Server 2.3. on two
servers and 20 clients.
Installed and configured Samba server, Webmin,
Squid server, Quota Allocation, Apache Web Server
including virtual hosting,Sendmail server including
masquerading, NFS server,NFS clients,DHCP Server
and DNS server.
Installed JAVA, PHP3 and Star office.
Installed Oracle 8 on one server.
Secured the network with complete network security
planning and implemented the security plan. Setup the
cheops for network monitoring and availability of
clients.

2000 [November] - 2002 [May]
Worked as Systems Administrator/System Head in Sip Technologies and exports Ltd.
SIP Technologies is the business partner of Sun Microsystems, USA, KKL of Switzerland (KKL is the biggest nuclear plant in Switzerland), Zaplet Inc, USA.
I manage 253 systems, routers, point to point connectivity through International Private Leased Circuit, leased lines, dialup ISDN line, switches, Firewall, Mail server and proxy servers.
Responsibilities include installing Solaris O/S (2.6, 2.7 and 2.8) on Sun Sparc machines (Ultra Sparc 1, 2, 5, 10, 65, Netra and 450 Enterprise Server) and on Intel Machines, Installing Windows Powered, Windows 2000, Windows NT, Windows 95/98//Mellinium, Installing Linux (Caldera, Red Hat, SuSe, Manderake and Turbo), Installing Configuring proxy servers, web servers, oracle, NFS servers and clients, DHCP servers and clients, Mail servers and clients, application servers, backup, trouble-shooting the startup, network related problems, printing problems, maintaining AIX systems on IBM/RS6000 machines and regularly taking backup and other system administration related activities. Coordinating with the vendors, VSNL, Dishnet. Documentation.
* Sun Solaris and Related Activities
Installing different versions of Sun Solaris i.e. 2.6, 2.7 and 2.8 on Ultrasparc machines (Ultrasparc 1, 2, 5, 10, 65, Netra and 450 Enterprise Server). Installation and configuration of NFS server and clients, DNS server and clients, DHCP clients, Printing services through SAMBA and manually configuring the printing services, Trouble-shooting the startup problems and all the system related problems, installing and configuring mail servers (Sendmail and Lotus Dominos mail server), Application server (I-Planet), Web servers (Apache), Backup, Reprogramming the MAC address on the locked PROM's. Adding packages and patches and different software. Monitoring the critical systems for network intrusion, stability etc.
* Linux and Related Activities
Installing different versions of Linux (Caldera, Red Hat, SuSe, Mandrake, Turbo), Installation and configuration of NFS server and clients, DNS, DHCP server and clients, Printing services through SAMBA and manually configuring the printing services, Trouble-shooting the startup problems and all the system related problems, installing and configuring mail servers (Sendmail and Lotus Dominos mail server), implementing Mailman, Web servers (Apache), Backup, Monitoring the critical systems for network intrusion, stability etc. Compiling the kernel, updating the kernel with the latest patches. Installing new embedded Linux kernel on Zarus.
* Windows and Related Activity
Installing different versions of Windows (Powered, 2000, NT, Windows XP, Millennium, 98, 95). Installation of Oracle, DHCP servers, Proxy servers, Mail server (Lotus Dominos), Print server, Defining system policies for users, installing application server (I-Planet), implementing Trust Relationship between multiple domains, backup and other system administration related activities.
* Routers and Switches
Installed and configured Cisco routers 2500 and 1750. Configured 1750 for IPLC and VoIP, Configured 2500 for two leased lines. Implemented ACL on 2500.
Configured Cisco 3500 10/100 MBPS switch, Nortel networks 10/10 MBPS switches. Installed a Linux based router and a SMC Barricade NAT router as intranet routers for splitting the networks to improve the network performance.
* Firewall
Installation and configuration of Sunscreen firewall on Solaris 2.6 and Linux native firewall.
* Mail Server
Installed and configured Sendmail on SunSparc & Linux for the main mail servers and subsequent clients. Also installed and configured Lotus Dominos mail server on Linux, Windows NT, Sun Solaris as Intranet mail servers. Installed Mailman for mail filtering and complete control over inbound mails.
* Proxy Server
Installed and configured squid and wingate proxy servers. I.P and user name authentication implemented. Installed and configured Apache WEB server to act as a proxy.
* LDAP Implementation
Installed and configured LDAP for Centralized Authentication
And e-mail resolution. LDAP server is running on RedHat
Linux 7.0.
* Documentation
I prepare and compile the complete documentation for the system administration department. Further I update the documentation on a regular basis. I also make the security policy for our network.
* SSH
Installed SSH on Linux & Windows for CVS, Outlook Express, secure terminal access to remote client systems and secure file transfer.
* Eddie (Load Balancing)
Installed & configured Eddie on Linux 7.2 for WEB Server load balancing.
* Bug tracking
Installation of Bugzilla for bug tracking.
* Concurrent Versions System
Installation of CVS and creation of different CVS accounts for
Different projects. Installed WebCVS on Linux 7.2 for remote
clients to access the CVS.
* Apache
Installed Apache on Linux, Windows & Sun Solaris.
Implemented SSL for secure access by the remote clients.
* Remote Desktop Administration
Installing and configuring remote desktop tools like VNC,
NetOp, Hyena, Microsoft Client Services for performing remote
administration on remote systems located in Santa Calara,
USA.

2002 [June] - 2003 [August]
Worked as Systems Administrator/System Head in
Javaji System Solutions.
Javaji System Solutions is a Frankfurt based software development and service provider with branches in USA, UK, Bulgaria, India and UAE. Partnering with Real, it provides the state of the art streaming media solutions. We are into embedded system solutions and system security services.
My responsibilities include managing the intranet, mail server, web server, installation and troubleshooting of Linux (6.2, 7.0, 7.1, 7.3, Midori) Windows 2000 server and professional), compilation of kernel etc.
I have installed and configured Apache web server with SSL, mail man, mail server, Helix universal server, Linphone, Gnome meeting, Samba, NFS, router (Linux based), SSH, CVS, Bugzilla, Squid proxy server, PGP, Midori etc.
I was working on three major projects.
1) Midori Linux
2) Web casting
3) Intranet Development

1) Midori Linux: We are porting Midori Linux to a specific SBC factor system. Midori is essentially an embedded version of the popular Linux. We are in the testing stage of this product, which will do lots of things like web casting, home-alarming, VPN, Firewall, DHCP server and client. Email client, VCD Player, router, Mobile phone and lots of functions. I am working on the basic installation and porting of Midori to SBC system along with konqueror-embedded, Gnome meeting, Linphone, Apache, Web casting, VPN, Firewall, DHCP server and client and router.
2) Web casting: Using real's Helix Universal Server and other tools we are giving state of the art web casting solutions. Be it on demand or live. Till date I have installed and configured Helix Universal Server on Red hat 7.3 and given the solutions to one of the biggest government organizations. I have given the complete network layout and authentication mechanism along with the router level PPP authentication.
We also provide synchronization of audio, video and PPT clips either live or on demand.
3) Intranet Development: Installing, configuring and implementing mail server, DNS server, Intranet WEB site which includes mailing lists, web interface to the local e-mail accounts, bug tracking, web interface to the CVS server, web based e-learning.

SINCE 2003 [September]
Working as Tech. Support Manager /Systems Administrator in Elxlinux Pvt Ltd.
Elxlinux is a premier Indian Linux company which makes it's own Linux distro. It has branches in USA, UK, India and Malaysia.. We are also into embedded system solutions. My responsibilities include managing the Support Max system and also manage the power isp project. I also look at various system administrative activities which includes the typical system administrative activities like mail server, web server dns etc management.
Currently I am working on Power ISP.
This product is very unique in that it gives the administrator a complete flexibility in controlling the bandwidth and distributing it the way he likes. The system administrator can see complete reports like which user has gone to which sites and even when required he can see user's desktop remotely.
I am currently handling this project.