Help with shell script

[Dan Day]

I'm new to Unix shell scripts and I can't quite figure out how to put this one together. What I would like to do is create a script that will check every 5 minutes to see if a process is running and if it's not to restart the process. I know all the steps I need the script to take but I can't figure out how to word them. I assume the steps would be:

1. Use ps and grep to figure out if the process is active
2. If it's active then exit the script
3. If it's not active then run the command to restart the process

I'm stumped at the beginning trying to figure out how to test if a process is active. I know I can use ps and grep to see if the process is there but I don't know how to use the output in an if/then statement. Another problem is if the process is restarted it will have a new PID so I can't always test for a specific PID.

On a side note, can anyone recommend any good books on shell scripting? I'm excited by all the possibilites that shell scripting provides and would love to learn more.

Thanks!
Dan

[mervTormel]

something like this to get you started...

$ cat mpr

#!/bin/sh

# mpr - my process restarter  

processKey="/Volumes/chunder/xapps/[e]DexterJavaDog"     
napTime=600  # seconds

while :
do

  procFound=`ps wwax | egrep -i "$processKey"`

  echo $procFound

  [ -z "$procFound" ] && echo "procDied: restarting /fullPathTo/theCommand"

  sleep $napTime

done

note well:

a) it is extremely important that processKey be entirely unique, and that a character gets bracketed so that the egrep process looking for it isn't found in the ps, e.g.,

$ ps wwax | egrep -i "/Volumes/chunder/xapps/eDexterJavaDog"

519 std S 1:49.81 /System/Library/Frameworks/JavaVM.framework/Commands/java -cp /Volumes/chunder/xapps/eDexterJavaDog eDexterJavaDog
1447 std R+ 0:00.00 egrep -i /Volumes/chunder/xapps/eDexterJavaDog

add brackets:

$ ps wwax | egrep -i "/Volumes/chunder/xapps/[e]DexterJavaDog"

519 std S 1:49.87 /System/Library/Frameworks/JavaVM.framework/Commands/java -cp /Volumes/chunder/xapps/eDexterJavaDog eDexterJavaDog

note: no egrep process found because of the bracketed string

b) you should test this thingy's behavior thoroughly

c) to put this into production, remove the echo's and put your command to run after the && switch, and run as a background job as: % mpr &

d) this process does not die; it sleeps for $napTime seconds

questions?

[Dan Day]

It looks like it will do exactly what I need it to. I'll test it out and let you know how it goes. The part about getting the grep search returned when doing the ps had me stuck. How does bracketing the one letter make it so it doesn't return the grep?

Also, how does this line work:

[ -z "$procFound" ] && echo "procDied: restarting /fullPathTo/theCommand"

Is the part in brackets doing a test on the results of $procFound ?


Thanks a lot for your help! This helped clear up a few things about how scripts are written.

I'll keep you posted,
Dan

[Dan Day]

OK, I just tested it out using TextEdit as the application to watch and it worked like a charm. I removed the echo parts and ran the script with TextEdit open and it returned with $procFound and then I quit TextEdit and it restarted it for me. Excellent!

Is there any downside to leaving a process open like this all the time? Would it make a difference if instead of being a continual process I changed it to a normal script and had cron run it every so often?

Thanks again for your help!

[mervTormel]

Quote: How does bracketing the one letter make it so it doesn't return the grep?

magic ; given "string" grep won't find "[s]tring" because it doesn't contain "string"

hmm, harder to explain than to just "grok". the brackets are a regular expression notation that says "match one from a set"

the regexp to grep says to find an 's' in that string position, but the ps of the grep command doesn't have it because its string has brackets in it.

in other words, grep "[f]oo" file will not find string "[f]oo" in file because string "[f]oo" does not have "foo" in it.

Quote: [ -z "$procFound" ] && echo "procDied: restarting /fullPathTo/theCommand" Is the part in brackets doing a test on the results of $procFound ?

exactly. if the ps returns a null, the value of procFound is "" and the -z tests for length of string = zero

the && is a short circuit switch that says "execute comm1 AND comm2", the AND only passes control to comm2 if comm1 succeeds. so, if comm1 then comm2.

[Dan Day]

Ahhh..it's all coming together now! I love moments like this when everything clicks into place and I clearly understand what was just moments ago confusing the hell out of me.

I have one last question though, is it possible to issue more than one command if the process dies? It would be nice to not only restart the command but create a log entry somewhere saying when it was restarted. Can I just enter another command after the first or do I need a special character like the && after the first command?

[mervTormel]

Quote: Originally posted by Dan Day ...Is there any downside to leaving a process open like this all the time? Would it make a difference if instead of being a continual process I changed it to a normal script and had cron run it every so often?

downside? no. change it to a normal script? it is a normal script, so i don't understand that notion.

if you give it to cron, you lose control of it and you have the overhead of managing a cron job. and the overhead of a new process created every napTime interval. used to be that one of the most expensive things you could do is to create a new process. not too true anymore, but why start another process when just sleeping is adequate.

let's say, it's a background job under your mgt, and you quit TextEdit and you don't want it restarted. foreground mpr and kill it. you can't do that easily if it's a cron job.

but, you could enhance the script so you could talk to it as a cron job. you could have a sentinel file that alters the behavior of the script like so:

if [ ! -f ~/mprShutup ] ; then

  procFound=`ps wwax | egrep -i "$processKey"`

  [ -z "$procFound" ] && /fullPathTo/theCommand

fi

then, if you want to acquiesce mpr, you can

% touch ~/mprShutup

and to unleash it.

% rm ~/mprShutup

[mervTormel]

Quote: Originally posted by Dan Day I have one last question though, is it possible to issue more than one command if the process dies? It would be nice to not only restart the command but create a log entry somewhere saying when it was restarted. Can I just enter another command after the first or do I need a special character like the && after the first command?

sure. just turn the short circuit && into a full if-then statement...

procName="/pathTo/command"

procFound=`ps blah blah blah`

if [ -z "$procFound" ] ; then

  $procName

  logger -i -p local0.notice -t mpr "rejuvinated $procName"

# any number of more statements here

fi

[Dan Day]

Thank you so much for your time and help merv. You've explained everything very well and helped me learn more in a few posts than I did all morning digging through books.

Take care!
Dan

[mervTormel]

yer welcome, but don't throw your books out. unless they're crappy ones. then throw them out because they'll cause more trouble than they'll solve.

[THEM]

The shell variable $? contains the exit status of the last command, 0 for success and anything else for failure.

so if you do ps -xaww | grep -i $ProcName you can test the success of that command by the following

if [ $? -eq 0 ]
then
<stuff>
fi

or -ne for not equal.

[Dan Day]

No, I won't throw the books out. I started with "Sams Teach Yourself Unix in 24 hours" which was ok but obviously didn't go too in depth. Then I picked up "Mac OS X Unleashed" which is an excellent book but they only scrape the surface of shell scripts. We have a Solaris server here at work that I was learning on before I was able to get OS X. Since it's a server used for production I didn't want to test something like this out in case I screwed it up. Now that I have OS X I can start playing with Unix more and try out things I couldn't before. I'm actually going to use this script on our server as well. I already made the necessary changes to get it to work on Solaris and it worked fine.

Can anyone recommend any good books on shell programming? I searched Amazon and the only one that looked decent was "UNIX Shell Programming, Revised Edition" and that's from 1989.

[Dan Day]

Quote: Originally posted by THEM The shell variable $? contains the exit status of the last command, 0 for success and anything else for failure. so if you do ps -xaww | grep -i $ProcName you can test the success of that command by the following if [ $? -eq 0 ] then <stuff> fi or -ne for not equal.

Thanks for the tip. That's one thing I'm really starting to like about Unix is all the different ways you can accomplish the same goal.

[TheBear]

Hi all,

Using the bash shell, how can I:

1 - get the current date and manipulate it so that I can create a folder styled yyyymmdd?

2 - read the first 10 charactors on the second (or later) lines of a text file so that I can verify the date of the file - this is a data file that I use wget to download and then need to copy elsewhere into a folder I create based on the date of the data file - the file is a comma seperated file with the first line being the header and all other lines having data, the first column being the date the file was created styled yyyy-mm-dd.

Further, anyone have any recommendations for bash programming resources to help leanr? I found an online book, and it says it assumes no prior knowledge of bash or scripting, yet it doesn't even mention what I consdier simply things like gettting the date.

Thanks

Dale

[mervTormel]

1. the date command will allow you to craft the format of the date by specifying format strings '%x' . see man pages for date and strftime...

$ d8=`date "+%Y%m%d"`    # load d8 with yyyymmdd

$ echo $d8
20020618

$ mkdir /pathto/foo_${d8}_bar

2. there are several approaches that can be taken here depending on the format and consistancy of the files. since you specify 'second (or later) lines' it sounds like a hunt job for grep, but i'd have to see the head of a few files to know. so, post the head of a few files indicating what you're looking for...

$ head file1 file2 file3

as for bash programming resources, o'reilly's bash book, read lot's of good and bad code, google: [ unix shell programming ]

the date command isn't a bash-ism per se, it's a unix command available to all shells.

you should get familiar with apropos and man.

$ apropos date | egrep -i '\([18]\)'
822-date(1)              - Print date and time in RFC822 format
...
date(1)                  - display or set date and time
...

$ man date
... see also strftime(3) ...

the above egrep gnarly-ism will filter out the ugly lib entries for an apropos search, only considering commands (1) and admin commands (8)

[TheBear]

Hello,

Thanks for the reply. The part about secondline or later was not clear. I should have said starting on the second line... all lines after the first line contain the information I am looking for. So I would need to be able to read those first 10 charactors on any line after the first.

Thanks

Dale

[mervTormel]

dale, still can't help you craft the magic for the files.

scripting is like a baking recipe; and one needs to be very intimate with all the ingredients and steps.

will need to see some representative files and the explicit steps you want to perform. let's call it pseudo-code. you enumerate what you want to do, in english, and that can be crafted into a sh script. e.g.,

read a file, read line two, get the first element delimited by comma, it's gotta be a date > 20010101, make a directory named foo_thatDate_bar, read the rest of the file? do we need to read more that the second line of the file? move file to dir foo_thatDate_bar, lather, rinse, repeat.

[TheBear]

Hello,

Quote: Originally posted by mervTormel dale, still can't help you craft the magic for the files. scripting is like a baking recipe; and one needs to be very intimate with all the ingredients and steps.

Quote: will need to see some representative files and the explicit steps you want to perform. let's call it pseudo-code. you enumerate what you want to do, in english, and that can be crafted into a sh script. e.g.,

Here is a snippet of one of the files I need to read the date from. The field names on the other files are different, but the date field is the first field.

Date,ISIN2,ISIN,cpCode,trCode,Name,CPToday
2002-06-18,DE0007931966,DE0007931974,IXBA,IXBB,oxx £ Overall,103.7678379128
2002-06-18,DE0007931982,DE0007931990,IXBC,IXBD,oxx £ Overall 1-3

Quote: read a file, read line two, get the first element delimited by comma, it's gotta be a date > 20010101, make a directory named foo_thatDate_bar, read the rest of the file? do we need to read more that the second line of the file? move file to dir foo_thatDate_bar, lather, rinse, repeat.

right, I download the files using wget and place them in a specific folder on the Mac. I then would like to read the first 10 chars of the second line of the file (this is a key thing that I do not know how to do, or where to look to find out how to do it).

parse those 10 charactors and then create a folder with that date on the remote server

copy the file(s) to the proper folder on the remote server.


The parts that I do not know how to do is reading the file and grabbing on the first 10 chars on the second line. Would using tail -1 be a good start to just grab a single line to work with (all the dates in the same file will have the same date).

If using tail would be a good start, then how does one grab only the first xx charactors? Would that possibly be by using grep? (not a much of a grep user here, yet).

Then use grep again to pasre the year/month/day (these are separated by - in the file).

Then somehow build a value to create the folder (if it doesn't already exist) styled yyyymmdd which was taken from the info above.

and finally, move (or copy, it doesn't matter here) to the destination folder.

Does that sound reasonable?

I don't know how, using unix scripting of any kind, how to set variables like what I believe I need to do above. I use a (commercial) product called Webcatalog for most of my sites, and while there are some similarities, the differences are too big to just transfer over.

One of the problems with man files is not knowing which man file to be reading. I actually tried some of the examples in the date man page and they did not work. And yes, I used the man page on the same machine I tried it on.


Thanks again.

Dale

[mervTormel]

Dale, your inquiry about shell scripting begs some follow-up. if you’re intention is to do a lot of this kind of thing, i would be remiss in not prompting you to also examine perl scripting as a more justifiable alternative, your-expensive-time-wise. in other words, if you’re going to invest time in learning scripting, sh scripting is good to know, but as you get so far with it, it can sure compel you to learn perl.

so, with that said, i cobbled some sh script to get you started; it appears to work, but it makes a lot of assumptions. and there are better ways to do this, a good perl scripter could whack this out in about five lines. and this may convince you where to focus your efforts. perhaps a perl programmer will come along and show us the five lines to elegantly handle this.

the chicken-scratch below demonstrates sh scripting; bash is a superset of sh with a lot of interactive features and some of the niftier features of csh family.

#!/bin/sh

# cook files in /pathto/kettle/*
# serving them to /pathto/yyyy-mm-dd_bowl/

# let's use explicit command paths & switches

grep="/usr/bin/grep -Ei"
cut=/usr/bin/cut
mkdir=/bin/mkdir
mv=/bin/mv
dirname=/usr/bin/dirname

# craft the source and target filespecs
# this is a bit of a kluge. but, specify
# your dirs in the kettle and bowl dir
# references below

# echo the expanded wildcard of the files

files2cook=`echo ~/work/kettle/*`

# get the canonical dirname of the target

targetBowl="`${dirname} ~/work/bowl/`"

# display the files we'll be sauteeing

echo $files2cook

# cook the files

for zfile in ${files2cook}

do

# get the yyyy-mm-dd string from the second line 
# of the file, delimited by a comma

# we think all the files look like so:

# Date,ISIN2,ISIN,cpCode,trCode,Name,CPToday 
# 2002-06-18,DE0007931966,DE0007931974,IXBA,IXBB,...
# 2002-06-18,DE0007931982,DE0007931990,IXBC,IXBD,...

# and we want to get that date in the second line, so
# grep for "Date," in the header line and include the
# next line (-A1), then grep excluding "Date," (-v) and
# pipe that to cut to extract the first field delimited
# by comma

  d8str=`${grep} -A1 "^date," ${zfile} | \
    ${grep} -v "^date," | \
    ${cut} -d"," -f1`

# we made a lot of assumptions above, like, all the 
# files2cook are the same format, guaranteed, and there 
# are no other kinds of files in the mix. but, we had
# ought to do some checking here and bail if things 
# don't look kosher.

# echo ${d8str} | sed pattern match [yyyy-mm-dd] ?
# if [ ! $? ] ; then we don't have a good d8str, better bail

# display the date string we found

  echo ${d8str}

# now to serve the cooked files

# craft the target dir with the date string in it

  targetDir="${targetBowl}/${d8str}_bowl"

  echo "  # comment me out and the end echo quote below to really run

# if the target dir doesn't exist, make it

  [ ! -d "${targetDir}" ] && ${mkdir} "${targetDir}"

# cross your eyes and move the files

  ${mv} "${zfile}" "${targetDir}"

  "  # end echo quote

done

the actual business end of the script, mkdir and mv, are merely echo'd to the terminal, because they are surrounded by an echo " the business end "

that ought to keep you entertained for a while. use man pages { i know, groan }
to get a grep on some of the commands:

man sh
man grep
man cut
man dirname

[TheBear]

Merv,

Thanks for the wealth of info - it even makes a lot of sense I have been away for a few days and just getting back to things.

I scrapped what I had been working with (a combination of WebCatalog and bash scripting) and just made some changes to what you wrote.

Now I have only 1 other problem. I have created 2 bash scripts. The first one controls wget to download the files, the second one (which is what you wrote) is supposed to parse the info and copy the files to the remote server (and create a folder if it does not exist).

The problem I am having is permissions (it would seem). I can execute the pasre/copy script from the command line as me with no problems. Everything works fine, copys the files, makes directories if needed and so forth.

However, if I run it as a crontask (using CronniX to configure) as user dale it doesn't seem to work. If I run it from the command line as root it tells me permission denied. So it would appear that it has to be run as user dale to match permissions on the remote server.

Using CronniX to configure the cron task, I have told it to run as user dale, but it still doesn't work - no error messages either.

Any ideas? I have also confiugre sudousers with a line to allow user dale to execute these scripts, although that would seen to not be needed.

Thanks

Dale