Thursday, April 30, 2009

Basic python threads - host pinger

Just a basic script modified to be useful. I stole the bulk of it from http://stackoverflow.com/questions/316866/ping-a-site-in-python (sorry!).

As identified, the problem with basic shell scripts or sequential code to ping dozens of hosts is that there's no inherent threading. You could background the commands, log to files, concatenate the results and generate a report, but that's annoying.

Python has pretty easy threading let's have a go of that.

Create a file with a list of host names or IP addresses called "serverlist.txt".

Put the following code in a python script:

from threading import Thread
import subprocess
from Queue import Queue

listfile = "serverlist.txt"

num_threads = 4
queue = Queue()

ips = []
hostfptr = open(listfile)
while True:
hostname = hostfptr.readline().strip()
if not hostname: break
ips.append(hostname)

#wraps system ping command
def pinger(i, q):
"""Pings subnet"""
while True:
ip = q.get()
##print "Thread %s: Pinging %s" % (i, ip)
ret = subprocess.call("ping -c 1 %s" % ip,
shell=True,
stdout=open('/dev/null', 'w'),
stderr=subprocess.STDOUT)
if ret == 0:
print "%-10s Alive" % ip
else:
print "%-10s No response" % ip
q.task_done()

#Spawn thread pool
for i in range(num_threads):

worker = Thread(target=pinger, args=(i, queue))
worker.setDaemon(True)
worker.start()

#Place work in queue
for ip in ips:
queue.put(ip)

#Wait until worker threads are done to exit
queue.join()


You may need to adjust the ping parameters if "ping -c 1 " doesn't work on your OS. And assuming a unix-like ping that returns an exit code of 0 if the ping was a success.

Run the script and you should find it quickly prints the status of each host in your server list.

If you like, you can uncomment the "##print" in the pinger function, it will show which thread it's using (out of the 4 we've made available).

One thing I've noticed after running it a few dozen times is some thread errors when the interpreter is exiting:

Exception in thread Thread-2 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
File "/usr/lib/python2.5/threading.py", line 446, in run
File "./pingle2.py", line 23, in pinger
File "/usr/lib/python2.5/Queue.py", line 165, in get
File "/usr/lib/python2.5/threading.py", line 209, in wait
<type 'exceptions.TypeError'>: 'NoneType' object is not callable


Sounds like the threads aren't really complete before the script ends?

Just to test that theory I added an explicit test after the queue.join() which is supposed to wait until they're done:

if queue.empty(): print "good!!"
else: print "onoes!!"


Result: It reports "good!!" even when the error appears. So not really sure what's going on here. Possibly some buffering from the os call?

Probably should use python native code to build its own pings - will look into it.

Monday, April 20, 2009

Python 2.6.2 on SCO OpenServer 5.0.6a

Just a basic notice regarding Python 2.6.2 on SCO OpenServer 5.0.6a in case people are doing a search for it. Haven't seen any other reports.

The build was done based on my previous instructions on creating a working development environment.

Had to run --without-threads to start with.

/bin/bash
/usr/gnu/bin/tar zxvf Python-2.6.2.tgz
cd Python-2.6.2
CFLAGS=" -O2 -s " LIBS=" -lsocket " ./configure --without-threads
make
make install


Result is a python build as follows:

Failed to find the necessary bits to build these modules:
_sqlite3 bsddb185 linuxaudiodev
ossaudiodev sunaudiodev zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_ctypes _curses _curses_panel
_multiprocessing _socket _ssl
_tkinter mmap readline
select spwd



Test.


Python-2.6.2 >./python --version
Python 2.6.2
Python-2.6.2 > make test
...
./python -E -tt ./Lib/test/regrtest.py -l
Traceback (most recent call last):
File "./Lib/test/regrtest.py", line 168, in
from test import test_support
File "/u/he61041/source/Python-2.6.2/Lib/test/test_support.py", line 8, in
import socket
File "/u/he61041/source/Python-2.6.2/Lib/socket.py", line 46, in
import _socket
ImportError: No module named _socket
make: [test] Error 1 (ignored)
./python -E -tt ./Lib/test/regrtest.py -l
Traceback (most recent call last):
File "./Lib/test/regrtest.py", line 168, in
from test import test_support
File "/u/he61041/source/Python-2.6.2/Lib/test/test_support.py", line 8, in
import socket
File "/u/he61041/source/Python-2.6.2/Lib/socket.py", line 46, in
import _socket
ImportError: No module named _socket
make: *** [test] Error 1


Yep I would call it a showstopper if the socket lib fails.

Thursday, June 5, 2008

Adaptec 29320 ALP-R with SCO OpenServer 5.0.6a

Getting (specifically) the Adaptec 29320ALP-R PCI-X card to work with 5.0.6.

First issue taken care of - disable HostRAID as instructed by all drivers that are listed below.

Initial Problems

1. The SCO 5.0.6 driver for ad320 here:

ftp://ftp.sco.com/pub/openserver5/drivers/OSR506/btld/ad320/

Lists 29320ALP as supported (whether there was an ALP without the ALP-R I can't determine).

After installing, configuring a drive/tape, and rebooting, the device isn't visible. Driver failed.

2. Most of the time basic BTLDs work from 5.0.7 (as long as it doesn't have management stuff dependent on 5.0.7 kernel changes). So trying this one:

ftp://ftp.sco.com/pub/openserver5/drivers/OSR507/btld/ad320/

It's the same driver! No point trying.

3. Next guess is to check Adaptec's site. Taking a look at their download section for the 29320ALP-R and "SCO Unix":

http://www.adaptec.com/en-US/downloads/unix/sco_unix?productId=ASC-29320ALP-R&dn=Adaptec+SCSI+Card+29320ALP-R

Only 5.0.7 listed, this could cause some alarms, but it's worth a shot. Unfortunately this one breaks the kernel. If you don't know how to fix the .c and .h files in the kernel your only option now is to reinstall 5.0.6.

This makes you less likely to take chances if you don't have a fast recovery process. Luckily I have a VM sandbox to test driver installs first (which only proves that 5.0.6 will install/rebuild the kernel, not that the driver will actually work with the hardware).

4
. Poke around the SCO FTP site a bit (it's become very messy), and looks like something a bit newer:

ftp://ftp.sco.com/pub/openserver5/507/drivers/ad320_1.1/

The README notes 2004 instead of 2003 as the year of release, that's promising. But no mention of what changes were made that might make it more compatible. But this one works, hooray!

What was the problem?

Usually cards are listed exactly as they are called in drivers and PCI-X support in 5.0.6 was very limited even toward the end. My guess is it could be combination of newer ALP-R revisions of the 29320ALP card, perhaps the addition of the PCI-X bus with possibly a new device ID that wasn't recognised by previous drivers.

Now add user confusion with:

  1. SCO's FTP site doesn't have the working 5.0.7 driver in the 5.0.6 section of the FTP site.
  2. SCO's FTP site having a completely different base sub directory for a whole pile of updated drivers that list 5.0.6a as supported.
  3. Adaptec's support downloads for "SCO Unix" only list 5.0.7 files, which cause kernel rebuild errors (which must be fixed or reinstalled).

Solution

Install the ad320 1.1 driver from SCO's 5.0.7 separate driver section.

Either as the BTLD (link=ad320) or post-install as follows:

marry -a /tmp/ad320_btld.img
mount -o nolog /dev/marry/tmp/ad320_btld.img /mnt
/etc/btldinstall /mnt


Continue with mkdev tape/hd, whatever, reboot, and voila.

Hopefully this helps someone, good luck.

Saturday, May 24, 2008

Solaris 10 and VMWare Server

It seems that to install Solaris 10 (5/08 32bit) on VMWare Server (1.0.5 32bit), XP (SP2 32bit), you probably need >512MB RAM.

My first attempt to install with 512 (the "recommended" amount of memory according to Sun) it failed and halted during "determining size requirements".

Rebooted, adjusted to 756MB RAM, passed that stage with no problems. The rest of the installation was fine.

Problem #2: After rebooting, logging in, adjusting resolution (which doesn't stick after a reboot by the way). Mount the VM drivers, run the install script, use default answers for all the prompts. Reboot. Several unattended reboots later it finally comes back up and is extremely sluggish.

Oh well, using without drivers for now, seems fine interactively.

VMWare site down for maintenance?

Irony? www.vmware.com - a company specialising in products that help you keep 100% uptime is down "for maintenance" a lot of the time I access it to license/download.

IBM DSA get to the command line

IBM Dynamic System Analysis (IBM page) is just a Linux boot CD like most of their other utilities.

They don't seem to want to give you the idea you can transfer the diagnostic log straight over the network. The log suggest dhcp is "disabled" yet one of the options from the trapped menu is to "ftp" the file to IBM support.

ctrl-c, break, ctrl-d, etc have all been trapped.

One thing they forgot to do, prevent shelling out from the viewer - if you type "view" to view the log, you can type ":!sh" to spawn a shell and do whatever you want. ifconfig shows that you've already picked up a DHCP address and you're on the network.

Monday, May 19, 2008

Installing SCO OpenServer 5.0.6 with BTLD on CD

Scope: To install OpenServer 5.0.6 onto a server with no supported floppy drive (eg. servers that use wacky new USB emulation modes that 5.0.7 supports).

Initially the idea was from Bela Lubkin's suggestion on this usenet post. The concept is to build the BTLD into the El Torito filesystem of a recreated install CD so that a separate floppy is no longer required.

I emailed Bela to clarify a couple of things which he kindly replied to and provided some great tips. He now works for VMWare so he's also provided some tips that will come in handy for that too, I'll put some of his tips at the end of this post for those interested. The core instructions will also be cross posted to Usenet for long term archival/search.

What you will need:

  • SCO OpenServer 5.0.6 installation CD. It should be legal for someone to make you a copy if you don't have one, since you will still need license codes.
  • SCO OpenServer 5.0.7 July 05 ISO from here (definitely required on the new boot disk we're creating for some newer hardware/VMWare - see notes from Bela later).
  • The geteltorito perl script.
  • The BTLD floppy or image you're expecting to require (eg. ServeRAID ipsraid, LSI lsil, etc).
  • A working SCO box to mount the floppy images.
  • A SCO or some other *ix box that has mkisofs/cdrecord to prepare the new boot image CD.
  • Any additional drivers which might be convenient to add to the SCO boot CD so you don't have to scrap together all your drivers next time you reinstall.

Generic Procedure:

  1. Extract the 5.0.6 install CD el torito image => 506bootfd.img
  2. Extract the 5.0.7 install CD el torito image => 507bootfd.img
  3. Mount the boot images and the BTLD on an OpenServer box.
  4. rm /mnt506/unix.notebook (if you don't need it and need space for BTLDs)
  5. cp /mnt507/boot /mnt506/
  6. cp -pR /mntbtld/btldname /mntbtld/install /mnt506/ (for each BTLD you want to add).
  7. Optional: edit /mnt506/etc/default/boot to set your default boot string, or add new entries as aliases for multiple BTLDs already built into the boot string.
  8. unmount 506bootfd.img (and the others if you like)
  9. mount 5.0.6 boot CD (eg to /mnt506cd for this example)
  10. mkdir /tmp/506newimg
  11. cp -pR /mnt506cd/* /tmp/506newimg
  12. cp 506bootfd.img /tmp/506newimg (506bootfd.img = the modified one we added the BTLD to).
  13. Copy any other drivers you want, to /tmp/506newimg now - eg. NIC drivers.
  14. mkisofs -r -v -l -b 506bootfd.img -o 506customcd.iso /tmp/506newimg
  15. Burn 506customcd.iso (or virtual mount if under VMWare).
  16. Boot using this CD on the new server and boot string: "defbootstr link= btld=fd(64)ipsraid". With any additional strings as required to help SCO find the root disk.
  17. Install as normal, press enter when it asks you to put the floppy with the BTLD in the drive, near the middle it will attempt to install the BTLD permanently, this will fail, follow the prompts until you get the opportunity to "(b) abort BTLD load, but continue the install" - press "b".
  18. Reboot the server, keeping the same CD in the drive to boot from it (in VMWare don't forget to go into the virtual BIOS and update the boot order if you need to - it automatically reduces the CD boot priority once the MBR is modified).
  19. At the boot prompt use the boot string: "hd(40)unix root=hd(42) link=fd(64)ipsraid Srom=wd(0,0,0,0) Sdsk=ips(0,0,0,0)" Modifying BTLD names and parameters as required.
  20. At the single user login prompt, enter the root password.
  21. mount /dev/cd0 /mnt
  22. /etc/btldinstall /mnt - follow the prompts.
  23. /etc/conf/cf.d/link_unix
  24. umount /mnt
  25. eject cd
  26. init 6
  27. Server should boot by itself!
  28. Optionally install the extra drivers from the CD once the server is back up.
  29. Start installing patches and customising your box. (Also, Acronis works well with SCO to cut an image of the hard disk - compression doesn't work as well if you don't cut an image with a relatively clean/fresh install, you can fit a base install image in under 700MB).

Detailed Procedure

1. Extract the 5.0.6 install CD el torito image => 506bootfd.img

./geteltorito.pl -o 506bootfd.img /dev/cdrom
Booting catalog starts at sector: 193114
Manufacturer of CD: SCO Product Integration
Image architecture: x86
Boot media type is: 2.88meg floppy
El Torito image starts at sector 193115 and has 5760 sector(s) of 512 Bytes
Image has been written to file "506bootfd.img".
Notes: Parts 1 and 2 were done on a Linux box, since I don't have easy access to the CD drive of a SCO box.

2. Extract the 5.0.7 install CD el torito image => 507bootfd.img

./geteltorito.pl -o 507bootfd.img /dev/cdrom
Booting catalog starts at sector: 140269
Manufacturer of CD: SCO Product Integration
Image architecture: x86
Boot media type is: 2.88meg floppy
El Torito image starts at sector 140270 and has 5760 sector(s) of 512 Bytes
Image has been written to file "507bootfd.img".

3. Mount the boot images and the BTLD on an OpenServer box.

mkdir /disks
ftp/scp/rcp images into /disks
restore:/disks >ls -l 50?bootfd.img
-rw-r--r-- 1 root sys 2949120 May 23 09:31 506bootfd.img
-rw-r--r-- 1 root sys 2949120 May 23 09:31 507bootfd.img
mkdir /mnt506 <- mount point for 506 boot image mkdir /mnt507 <- mount point for 507 boot image marry -a /disks/506bootfd.img
marry -a /disks/507bootfd.img
mount -o nolog /dev/marry/disks/506bootfd.img /mnt506
mount -o nolog /dev/marry/disks/507bootfd.img /mnt507

restore:/disks >df -Pk
Filesystem 1024-blocks Used Available Capacity Mounted on
[...]
/dev/marry/disks/5 2880 2256 624 79% /mnt506
/dev/marry/disks/5 2880 2755 125 96% /mnt507
Key information: 624KB free on the floppy image to work with, plenty of space. Not much room on the 507 image, let's see what's on the filesystems.

restore:/disks >ls -l /mnt506
total 4200
-r--r----- 2 bin bin 52961 Jul 28 2000 ahslink
-r-------- 1 bin bin 74096 Jul 28 2000 boot
drwxrwxrwx 3 bin bin 48 Jul 28 2000 etc
-r--r----- 2 bin bin 52961 Jul 28 2000 link
drwxrwxrwx 4 bin bin 64 Jul 28 2000 opt
-rw-r--r-- 1 bin bin 150810 Jul 28 2000 rootFS
-rw-r--r-- 1 bin bin 1182756 Jul 28 2000 unix
-rwxr-xr-x 1 root sys 616630 Jul 28 2000 unix.notebook


You can delete the unix.notebook kernel if you need space on the 506 disk, but I don't think you'll need to for most.

restore:/disks >ls -l /mnt507
total 5190
-r--r----- 2 bin bin 52985 May 20 2005 ahslink
-r-------- 1 bin bin 74648 May 20 2005 boot
drwxrwxrwx 3 bin bin 48 May 20 2005 etc
-r--r----- 2 bin bin 52985 May 20 2005 link
drwxrwxrwx 4 bin bin 64 May 20 2005 opt
-rw-r--r-- 1 bin bin 154182 May 20 2005 rootFS
-rw-r--r-- 1 bin bin 1652840 May 20 2005 unix
-rw-r--r-- 1 root sys 648826 May 20 2005 unix.smallmem


If you're adjusting this procedure to install a BTLD on the 507 disk you probably want to make space by deleting the >600KB unix.smallmem.








TBC