major errors with "here documents" in bash
 

Well, this all started, when I was trying to compile sqlite 3.2.x
and run in on my MacOS X 10.3.9 machine. ( got compiling error )

Then is was trace to Shell problems, So I wipe the hard drive,
and re-install MacOS X 10.3.9 with all updates, and nothing else added.
( did this three times )
--

These test where run, to see if Shell is giving garbage characters,
which it is. ( again, this system is a virgin system with standard defaults of install)

<I will add this>
Last login: Sat Sep 10 12:47:53 on ttyp1
Welcome to Darwin!
G4:~ rn$ env
TERM_PROGRAM=Apple_Terminal
TERM=xterm-color
SHELL=/bin/bash
TERM_PROGRAM_VERSION=100.1.8
USER=rn
__CF_USER_TEXT_ENCODING=0x1F5:0:0
PATH=/bin:/sbin:/usr/bin:/usr/sbin
PWD=/Users/rn
SHLVL=1
HOME=/Users/rn
LOGNAME=rn
SECURITYSESSIONID=20ef80
_=/usr/bin/env
G4:~ rn$


--

Last login: Sat Sep 10 09:59:41 on console
Welcome to Darwin!
G4:~ rn$ cat << EOF > foo
> test 123
> EOF
G4:~ rn$ od -c foo
0000000 264 b 237 ^ 332 337 m \f =
0000011
G4:~ rn$

Other Test That I ran...

Last login: Sat Sep 10 11:52:38 on ttyp1
Welcome to Darwin!
G4:~ rn$ cat << EOF
> line 1
> line 2
> EOF
?b?^??m
=A??cG4:~ rn$
=========================
as for you UNIX guru's, you can see something is not right?
but, as of yet, have no one can nail it down to what is causing this
problem.

Machine: G4 sawtooth ,with upgrade CPU OWC G4 1.5 Ghz
2.0GB of RAM, adaptec controller 29160, ATI AGP video .
MacOS X 10.3.9 with all updates.

--

Thanks -

:confused:

It looks weird but I would try to narrow down the problem but doing some of the following:
- use the full paths to commands like 'cat'
- trying a different shell (e.g. tcsh)
- logging in as a different (preferably freshly created user - with no shell "dot" files)
- using an editor (.eg. 'vi' or 'pico') to create files and see if that works
- running the command 'stty sane' and then 'stty -a' to see what the settings are
- trying a different Terminal program (e.g. xterm under X11)
- remotely logging into this machine to see if that makes a difference

Got this in X11 window

G4:~ rn$ cat << EOF
> line 1
> line 2
> EOF
*"44¥/-Ãn+¥ÓG4:~ rn$

===
wait it get worst..

G4:~ rn$ cat << EOF | hexdump -C
> test 123
> EOF
00000000 a0 22 34 92 34 8b a5 2f 2d |."4.4../-|
00000009
G4:~ rn$



=====
where the test 123.????

So, there problem is in both OSX and X11 terminal windows, running bash


-====

NOW get this: running .tcsh
I get this !!

Last login: Sat Sep 10 21:02:37 on ttyp1
Welcome to Darwin!
G4:~ rn$ tcsh
[G4:~] rn% cat << EOF | hexdump -C
? test 123
? EOF
00000000 74 65 73 74 20 31 32 33 0a |test 123.|
00000009
[G4:~] rn%

So, there must be something wrong with BASH, but right?

Also, ran this test:

[G4:~] rn% cat << EOF
? line 1
? line 2
? EOF
line 1
line 2
[G4:~] rn%

So, it would appear that tcsh shell is running okay compare to bash,
but where is bash mess up, is there corruption on the apple panther install disk.. or ????

You didn't seem to try all of what I suggested above - in particular the suggestion to try a different user (to avoid the influence of your "dot" files).

Yes, I made a new users, test with no admin priv
and still the problem with there in bash.

Richard

I assume you didn't create "dot" files for that new user.
Have you possibly modified /etc/profile or /etc/bashrc or anything else that might affect bash?

Another thing to try:
Create a new sub-shell by typing:
/bin/bash
at the prompt. Does the problem occur in this sub-shell?

I did a trace between two shell windows, and a kdump
to see where it breaks, several other friends chime in on this,
and one is finding where it breaks, here is a <snip> of that email.

--

Richard wrote:
[]
> Here is the output,
> I saw (2) -1
> But why are there other pid listed? since the second terminal was 508
>
> "508 bash RET wait4 -1 errno 10 No child processes"
> "508 bash RET stat -1 errno 2 No such file or directory"

These are completely normal. As is everything else in your ktrace,
except that it looks like the temporary file /var/tmp//sh-thd-1126424753
changes its content between the time it is written and the time it is
read again (strongly radioactive environment perhaps?). The only anomaly I can see is that in the unlink call

> 512 bash CALL unlink(0x809600)
> 512 bash NAMI "/var/tmp//sh-thd-1126424753"
> 512 bash NAMI "/var/tmp//sh-thd-1126424753"
> 512 bash NAMI "/var/tmp//sh-thd-1126424753"
> 512 bash RET unlink 0

the filename is repeated three times. There is only one of these when I
try the same. Everything else, except of course the crazy result

> 512 cat GIO fd 0 read 14 bytes
> "\240"4\M^R4\M^K\M-%/-\M-Cn+\M-%\M-S"

of reading the temporary file on file descriptor 0, is identical in my case. Maybe a weird bug in the unlink(2) system call?

=========== </snip>
As you can read, there is something not right with bash,
and all bash related shells... .csh and tcsh shell all works normal.

And YES, I did do a /bin/bash subshell, and the problem was still there
doing this command:
cat << EOF
line 1
line 1
EOF

instead of getting
line 1
line 2
I got garbage characters..

TKS -
Rick

:confused:

1) Especially when troubleshooting a strange issue like this one, it is important to report exactly what you have done and what results you got.
You failed to do so in your latest posting about using 'ktrace'. It isn't clear what command you were tracing. The reference to temporary files makes it seem that you were perhaps redirecting the 'cat' output to a file like in some of your previous examples, but not your most recent ones.
It would be best to keep to one example so you are sure you are trying exactly the same thing in all cases.

2) You say the problem afflicts all "bash related shells". Which shells do you mean? What have you tried?

3) I think it quite unlikely that there is any "weird bug" in any of the system calls. It is much more likely that your system has either some weird configuration, or else some of the executables or libraries have been corrupted somehow.

Let's start over and go slowly.
Are you experiencing this behaviour on a freshly installed copy of OS X - before you have installed any of your own files or changed any configuration?
Did you choose English as the language when you installed OS X?
Is your Mac a standard US model (as opposed to one with a non-US keyboard etc)?

What happens if you uninstall/disable the 3rd-party CPU upgrade ?
Same thing for the video card (assuming that it didn't come standard with the machine)?

Yes, on a virgin system ( in -fact ) 3 times, using a virgin system
with a fresh installed of 10.3.9 and its updates with NOTHING ELSE installed, and English is the language.. gotten these error.

The CPU is upgraded to a G4 1.5 Ghz from OWC from a G4 450 sawtooth.
No, I don't have the original CPU.
There is a adaptec scsi card running the main hard drive for OSX
there is also a ATI video (32 megs )agp slot.
there are two other IDE hard drives one for OS9,
and one for storage.
----
Rick

Since you weren't completely explicit, I have to ask again:
Did you do any customizing of the user account (e.g. copying over some files from another machine) after you installed (which created a fresh user account for you to use)?

1) When did you first notice this problem? I.e. what has changed? Hadn't you used the shell before this?

2) Just in case it is hardware related (a long shot) you should remove all non-essential hardware - e.g. the extra hard drives. And (if possible) remove the video card (substituting the original video card).

3) You should try booting into single-user mode to see if the problem exist there:
http://docs.info.apple.com/article.html?artnum=106388

NO (again) 3 times, did this standard install,
NOTHING ELSE installed (.)


It all started with trying to compile SQLite 3.2.x when I notice,
that C could not output executablies.

Then we ( other techie's ) started with CAT command, to find
that there where some problem with BASH.

CAN'T

Try that, root is read only,
CAT command can't write to file.


:(

You don't seem to understand that I am trying to get you to be a bit more voluble. We don't charge by the word, you know.
When people talk about "installing" something, they usually mean applications etc. And so I thought maybe you might have set up your user account somehow without thinking of it as "installation".

And you had never used the bash shell before that? Or merely not noticed any problem?

First off, it isn't necessary to write to a file in order to demonstrate the problem. From what you have said earlier, the problem evidences itself with just the 'cat' command with output to the screen:

% /bin/cat << EOF
line 1
line 2
EOF

And if you want to write to files in single-user mode, you can mount the disk as writable with the command:

/sbin/mount -uw /

As for users, just me (Rick) with admin priv.

second for bash, I have used bash shell doing simple network checking,
or log cleaning ... nothing above the very basic,
used pico a few times.. but that it.

it was only when I try to compile sqlite, when I saw errors,
in the terminal window, and in the config.log file.

Then started, to find the problem, posted this problem, on several list,
and got some good feedback.. the last feedback I got back was from,
Chet Ramey ( MR. Bash ) himself who wrote this:

"find out why it's apparently corrupting here documents"

Is that the only place where you see the problem?
I had thought that was merely an easy way to illustrate the problem but that it was more widespread.

What about the answers to my other questions?
1) What other "bash related shells" have you tried?
2) Does the problem evidence itself in single-user mode?

And here's another thing you can try that might help to narrow down the problem: mount your Sawtooth's disk on some other Mac and then run bash from your Sawtooth's drive (but executed on the other Mac) - e.g. via the full path /Volumes/name_of_drive/bin/bash

1. I've test all shells, shell WITH the problem are;
.bash .sh . zsh.
Shell that DON'T have any problem are;
.csh .tcsh
Running the SAME command test.

2. Not Yet.

3. Sorry, one Mac per customer. ( wish I have two.)

-R

Note that bash and sh are the same shell on OS X
(look at the results of ls -l /bin/*sh)

Could you please show what that test is - for future reference?
You have shown several different versions of tests in the above posts. In particular, it is interesting to know whether you are writing to a file or not.

And you didn't answer my question about whether the use of "here" documents is the only thing that gives trouble.

Hopefully soon - testing what happens in single-user mode may be quite useful for diagnosis.

I was thinking that you would use a friend's Mac. And the Macs don't have to be in the same room - you could mount a disk over the Internet.

Here are the shells...

Last login: Sun Sep 11 23:53:34 on ttyp1
Welcome to Darwin!
G4:~ rn$ ls -l /bin/*sh
-rwxr-xr-x 1 root wheel 603488 8 Sep 23:19 /bin/bash
-r-xr-xr-x 1 root wheel 348068 8 Sep 23:19 /bin/csh
-r-xr-xr-x 1 root wheel 603488 8 Sep 23:19 /bin/sh
-r-xr-xr-x 1 root wheel 348068 8 Sep 23:19 /bin/tcsh
-rwxr-xr-x 1 root wheel 479120 8 Sep 23:19 /bin/zsh
G4:~ rn$
--------

Second:
Here are some of the command that failed:

cat << EOF > foo
test 123
EOF

then do: od -c foo

---

cat << EOF
line 1
line 2
EOF

--
cat << EOF | hexdump -C
test 123
EOF

Again, thinking heredoc is broken.


-R

What happens if you just do:

% cat > foo
test 123

(where you type control-D to indicate the end of input)
This tests keyboard input without making use of the shell's heredoc syntax.

And it would be interesting to know if the output (e.g. from 'hexdump -C') is always the same in those cases where the problem occurs. I.e. if you always enter the one line:
test 123
do you always get the same output from 'hexdump -C'
Or is it random - varying each time in spite of the same input?
If it is always the same, maybe we could get a clue from the output what is happening. E.g. try it with the line: "test 123", then with the line "test 124", etc.

This is what happen: Nothing
G4:~ rn$ cat > foo
test 123
G4:~ rn$

Type cat > foo ( then return)
(type) test 123 (then return)
(type) control -D


Here is one hexdump

G4:~ rn$ cat << EOF | hexdump -C
> test 123
> EOF
00000000 d0 eb 55 ed 21 d8 06 e7 85 |..U.!....|
00000009
G4:~ rn$

second hexdump

G4:~ rn$ cat << EOF | hexdump -C
> test 123
> EOF
00000000 d0 eb 55 ed 21 d8 06 e7 85 |..U.!....|
00000009
G4:~ rn$

third hexdump

G4:~ rn$ cat << EOF | hexdump -C
> test 124
> EOF
00000000 d0 eb 55 ed 21 d8 06 e7 85 |..U.!....|
00000009
G4:~ rn$

and last !!

G4:~ rn$ cat << EOF | hexdump -C
> test 1234567890
> EOF
00000000 d0 eb 55 ed 21 d8 06 e7 85 e4 fd 7d dd 92 cc 46 |..U.!......}...F|
00000010
G4:~ rn$


-R

Just for the FYI.
I'm going to wipe the hard drive, and try a experiment....

when installing: macos x 10.3, I going to halt
the update process, right there... and try running CAT command.
this way, I will know if any corruption came off the disk,
some lib ?

Now, if cat << EOF works... then we can say, it must be the combind,
download to update to 10.3.9 that has a problem..
so I will break it down, until it breaks.

or,
if the cat command is broken right at the start, then
I need to call Apple.

Regards -
Rick
:cool:

Umm, you were supposed to look at the contents of the file "foo" to see if it got what you typed.

hexdump -C foo

1) This sounds like an interesting plan - it will tell you if the problem arose with a particular OS X update.

2) But before you do that, please boot into single-user and see if the problem exists there.

3) I don't think your hardware is supported by Apple, so I don't imagine you will get much joy that way. Besides which, it is out of warranty so I think you will have to pay to get tech support from Apple.

4) Speaking of the hardware, it would be nice to rule that out as the problem - that is why I suggested mounting your disk on some other Mac and running your copy of bash on that other Mac.

Well, I was right, :D
The cat test work just fine on 10.3 with NO updates...
running in bash

ie.
Last login: Mon Sep 12 14:33:41 on ttyp1
Welcome to Darwin!
G4:~ rn$ cat << EOF
> line 1
> line 2
> EOF
line 1
line 2
G4:~ rn$ cat << EOF | hexdump -C
> test 123
> EOF
00000000 74 65 73 74 20 31 32 33 0a |test 123.|
00000009
G4:~ rn$

G4:~ rn$ bash --version
GNU bash, version 2.05b.0(1)-release (powerpc-apple-darwin7.0)
Copyright (C) 2002 Free Software Foundation, Inc.
G4:~ rn$


=========
as you can see...
my guess, there a corrupt lib in the combind update (10.3.9)
I will download the combine to 10.3.3 and test that first,
see where it breaks.

We know, for a fact ,if I did a full updated, via software update,
it will break ALL the time. So again, something in the FULL 10.3.9 update,
is corrupted.

-R

:D