The macosxhints Forums - bash script troubleshooting

The macosxhints Forums (http://hintsforums.macworld.com/index.php)

- UNIX - General (http://hintsforums.macworld.com/forumdisplay.php?f=16)

- - bash script troubleshooting (http://hintsforums.macworld.com/showthread.php?t=40976)

mac-bonehead

06-15-2005 03:24 AM

bash script troubleshooting

I'm trying to get the following bash script to run, but keep getting this error:

awk: illegal field $(), name "n"
input record number 1, file ~/Desktop/myConverter/drop/inputFile.txt
source line number 1

Any suggestions on how to get this to work, or for making this cleaner, are welcome.

Thanks!

Code:

#!/bin/bash



# conversion script

# first assign some variables:



scriptPath=~/Desktop/myConverter

sourceFile=$scriptPath/drop/inputFile.txt

lineCount=`cat $sourceFile | wc -l`

echo "line count is "$lineCount

n=1



export scriptPath

export sourceFile

export lineCount

export n





# then the parsing of the file:



while [ $n -le $lineCount ]

        

        do

        

                # check for empty lines first

                

                if [ -n `awk 'NR==$n' $sourceFile` ]; then

                        

                        # create file with name from field 4 + '.ttg' suffix:

        

                        destFile=`awk 'NR==$n' $sourceFile | cut -f 4`.ttg

                        echo "destFile is now "$destFile

                        

                        sleep 3

                        

                        # output the following to the file:

                

                        # textBlock1

                        cat $scriptPath/textBlock1.txt > $scriptPath/pickup/$destFile

                

                        # field3 of line n

                        awk 'NR==$n' $sourceFile | cut -f 3 >> $scriptPath/pickup/$destFile

                

                        # textBlock2

                        cat $scriptPath/textBlock2.txt >> $scriptPath/pickup/$destFile

                

                        # field2 of line n

                        awk 'NR==$n' $sourceFile | cut -f 2 >> $scriptPath/pickup/$destFile

                        # textBlock3

                        cat $scriptPath/textBlock3.txt >> $scriptPath/pickup/$destFile

                

                        # field4 of line n

                        awk 'NR==$n' $sourceFile | cut -f 4 >> $scriptPath/pickup/$destFile

                

                        # textBlock4

                        cat $scriptPath/textBlock4.txt >> $scriptPath/$destFile

                

                        # field5 of line n

                        awk 'NR==$n' $sourceFile | cut -f 5 >> $scriptPath/pickup/$destFile

                

                        # textBlock5

                        cat $scriptPath/textBlock5.txt >> $scriptPath/pickup/$destFile

                

                fi

                

                # increment the line number

                n=$((n+1))

                        

                echo "N is now "$n

        

        done

hayne

06-15-2005 04:11 AM

I guess the problem is with the lines that have:
'NR==$n'
since shell variables are not interpolated (substituted for) inside single quotes.

And it seems to me that this script will run fairly slowly since it seems to invoke several processes per line. I hope there aren't too many lines in the source file.
I would suggest learning Perl and using that - it would be much easier (Perl is designed for text handling) and faster executing.

mac-bonehead

06-15-2005 11:55 AM

Thank you for the feedback.

It seems odd that bash wouldn't be able to handle such a simple task. And the source file is about 30 lines.

Are you saying perl is just a little easier, or am I really making it a pain by using bash?

Would python be better than perl if I'm going to try to learn a new language?

nkuvu

06-15-2005 12:31 PM

Quote:

Originally Posted by mac-bonehead

Would python be better than perl if I'm going to try to learn a new language?

Python and Perl are both pretty comparable in their capabilities. But just so you know, Perl vs Python is a Holy War in and of itself. (As in, there has been a lot of debate from either side of the fence as to which is a "better" language)

I personally prefer Perl, but that's mostly because I know Perl. I learned Perl due to the fact that I took over someone else's script at work -- otherwise I would have tossed a coin between the two.

Perl's origins lie with sed and awk and other tools, so that might be closer to what you're doing in bash. I don't use bash scripting (or sed or awk) at all, so I'm not really sure what this script is doing. But with a bit more description of your goals I'd be happy to help write a Perl script for this. I'm sure someone else could help with a Python script if that's the way you want to work it.

vancenase

06-15-2005 01:27 PM

i've (loosely) picked up both perl and python. i currently use python, but each has it's own merits and downfalls. i find it useful to know both! :)

if you give a description of what your script is supposed to do, and an example input file, i (or someone else) could show you can example of how to accomplish it in perl/python.

mac-bonehead

06-15-2005 01:30 PM

Thanks all - I'm a perl convert. Here's the beginning of the new perl script:

Code:

#!/usr/bin/perl



scriptPath = "~/Desktop/myConverter";

sourceFile = "$scriptPath/drop/myFile.txt";

lineCount = "`cat $sourceFile | wc -l`";

print "line count is " . $lineCount;



# then the parsing of the file:



for ($n=1; $n <= $lineCount; $n++) {

        

                # check for empty lines first

                

                        if (open($sourceFile, " | awk 'NR==$n'")) {

                

                        # create file with name from field 4 + '.ttg' suffix:

        

                        destFile = open($sourceFile, " | awk 'NR==$n' | cut -f 4".ttg);

                        

                        print "destFile is now " . $destFile;

                        

                }

                

                print "N is now " . $n;

        

}

So I'm just trying to get perl to verify that each line of the input file (sourceFile) isn't blank, then, for each line in the source file, assign an output filename (destFile) based on the contents of the 4th field of that line, then (haven't written this part yet), spit out various contents of each line into the destFile

I'm getting this error:

Can't modify constant item in scalar assignment at ./myScript.pl line 3, near "'~/Desktop/myConverter';"

Any ideas here?

hayne

06-15-2005 02:42 PM

Here's a first cut at a Perl script for you - completely untested.

Code:

#!/usr/bin/perl -w



my $scriptPath = "~/Desktop/myConverter";

my $sourceFile = "$scriptPath/drop/myFile.txt";



# then the parsing of the file:

open(SRC, $sourcefile) or die "Can't open $sourcefile: $!";

my $n = 0;

while (<SRC>)

{

    ++$n;



    # check for empty lines first

    next if (/^$/);



    my @fields = split;

    my $destFile = $fields[4].".ttg";

                        

    print "destFile is now $destFile\n";        

    print "n is now $n\n";

}

close(SRC);

Notes:
1) I added the "-w" to the first line. This makes Perl tell you when you are making silly mistakes.
2) Perl variables need to have a "$" (or other symbol) in front of their names - even when assigning to them.
3) You usually want a "\n" at the end of a print statement.
4) Best to open the sourcefile at the beginning and then process each line.
5) You really need to get a book on Perl. It will be too frustrating to try to learn by trial and error. The usual recommendation is either "Learning Perl" (the Llama book) or "Programming Perl" (the Camel book) - both from O'Reilly.

vancenase

06-15-2005 02:57 PM

i started a python one for you, but i won't post it unless you really want to see it :)

mac-bonehead

06-15-2005 03:16 PM

Wow - thanks a lot for writing that.

Just wondering what does the 'my' do before each variable assignment?

hayne

06-15-2005 03:19 PM

Quote:

Originally Posted by mac-bonehead

Just wondering what does the 'my' do before each variable assignment?

It makes the variable localized to the current scope - so it doesn't conflict with other variables of the same name at higher scopes. Not necessary - just a good practice.

mac-bonehead

06-15-2005 09:41 PM

Just FYI, the script you posted spits back the following error:

Use of uninitialized value in open at ./myScript.pl line 7.
Use of uninitialized value in concatenation (.) or string at ./myScript.pl line 7.
Can't open : No such file or directory at ./myScript.pl line 7.

Now to see if I can figure out why...

vancenase

06-15-2005 09:48 PM

not sure if this is the problem, but maybe change this

Code:

my $scriptPath = "~/Desktop/myConverter"

to the full path:

Code:

my $scriptPath = "/Users/USERNAME/Desktop/myConverter"

-- i had to do this for the python script i started

mac-bonehead

06-15-2005 09:55 PM

vancenase - tried switching to the absolute path as you suggested, still no love. Would like to see your Python script, if you feel like sharing...

hayne

06-15-2005 10:09 PM

Quote:

Originally Posted by mac-bonehead

Line 7 is this:

Code:

open(SRC, $sourcefile) or die "Can't open $sourcefile: $!";

and if you look closely enough, you will see that it is a simple typo.
The variable that was initialized was "$sourceFile", but line 7 is using "$sourcefile" (lower-case f).
If you now look again at the error messages, you will see that they are telling you what is wrong as well.

nkuvu

06-16-2005 12:19 PM

If you're using "my" declarations you might as well go all the way and use strict

Code:

#!/usr/bin/perl -w

use strict;



my $scriptPath = "~/Desktop/myConverter";

my $sourceFile = "$scriptPath/drop/myFile.txt";



# then the parsing of the file:

PrivoxyWindowOpen(SRC, $sourcefile) or die "Can't open $sourcefile: $!";

my $n = 0;

while (<SRC>)

{

    ++$n;



    # check for empty lines first

    next if (/^$/);



    my @fields = split;

    my $destFile = $fields[4].".ttg";

                        

    print "destFile is now $destFile\n";        

    print "n is now $n\n";

}

close(SRC);

run this script and it tells you Global symbol "$sourcefile" requires explicit package name at script_filename.pl line 8.

Which is what hayne just said.

The use of strict means you have to put "my" in front of your locally defined variables -- so it will catch errors such as typos in variable names. It does a whole lot more, but that's the bit relevant to this.

mac-bonehead

06-16-2005 03:00 PM

Cool - thanks a lot guys. Fixed the typo, and tried the path change idea (above), but still get:

Can't open file No such file or directory at ./myScript.pl line 8.

Have double-check and triple checked the paths.
Is it bad to combine two perl variables and embed them as I'm trying to do, in order to create the path?

Does perl not like the '~' character, or does it need to be escaped?

nkuvu

06-16-2005 04:12 PM

Quote:

Originally Posted by mac-bonehead

First, Perl works fine with multiple variables creating another. Consider the following snippet:

Code:

$some = "/Users";

$sort_of = "/nkuvu";

$pathname = "/Scripts/Perl/yay.pl";



$full_path = "$some$sort_of$pathname";



print $full_path;

prints /Users/nkuvu/Scripts/Perl/yay.pl

You don't need to escape the ~ character, but Perl might not be able to determine the path based on that. Edit: see below

If the path is correct and the file exists just fine, you may want to check permissions. Given the fact that you're using PrivoxyWindowOpen, I'm guessing that you may be using this as a CGI script of some sort. Yes? If so, you'll want to look into permissions and make sure your webserver can read the file. By default Apache runs as user www in group www on OS X. So if your directories have read and execute bits set for either group www or 'other' then you should be OK.

If that part doesn't make sense, take a quick peek at the Unix FAQ.

Edit to add:
Perl doesn't expand the ~ into a real directory, apparently. I tried the following commands:

Code:

Aliens:/ nkuvu$ touch ~/foo.txt

Aliens:/ nkuvu$ perl -e 'print "yay\n" if -e "~/foo.txt"'

Aliens:/ nkuvu$ perl -e 'print "yay\n" if -e "/Users/nkuvu/foo.txt"'

yay

So you may want to change the ~ to /Users/your_username in the script. The potential issue about permissions still applies, though.

hayne

06-16-2005 11:15 PM

Quote:

Originally Posted by nkuvu

Given the fact that you're using PrivoxyWindowOpen, I'm guessing that you may be using this as a CGI script of some sort.

I don't think that "PrivoxyWindowOpen" was supposed to be there. I think it was an artifact from the use of the "Privoxy" web proxy - it substitutes "PrivoxyWindowOpen" for "open" calls in order to avoid the worst of JavaScript abuses.

hayne

06-16-2005 11:18 PM

Quote:

Originally Posted by mac-bonehead

Cool - thanks a lot guys. Fixed the typo, and tried the path change idea (above), but still get:

Can't open file No such file or directory at ./myScript.pl line 8.

Note that there were two typos - the variable occurs two places in that line. It seems that you didn't fix the second occurrence - that is why we don't see the value of $sourceFile being printed in that error message (after the "Can't open file ").

You should follow nkuvu's suggestion and put
use strict;
as the second line of the script.

nkuvu

06-17-2005 12:23 AM

Quote:

Originally Posted by hayne

Aha!

I've been seeing this substitution on a number of posts at Perlmonks, and I was wondering where it came from. But I hadn't actually gone in and tried to find out what was going on.

Bonus cookies for hayne. Thanks.

CPYCGerry

06-21-2005 12:36 PM

Perl - a great IDE

Just in case:
I found a great IDE that makes developing Perl scripts a breeze.
Affrus available at: http://www.latenightsw.com/affrus/index.html

The price is high. But I've easily saved development time.

dr_pardee

11-19-2008 07:11 PM

It can be done in awk

If others don't have immediate time to learn Perl or Python, you could do this:
`awk -v varN=$n 'NR==varN' $sourceFile`

All times are GMT -5. The time now is 06:21 PM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
Site design © IDG Consumer & SMB; individuals retain copyright of their postings
but consent to the possible use of their material in other areas of IDG Consumer & SMB.