Downloading a URL to a Script variable (curl failing)
 

This AppleScript should download the HTML code from a URL (equivalent of "View Source" in a browser) to a variable in my script (the_HTML). It works for at least two other people on their own machines running OS X 10.3.8, but not for me.

set the_HTML to my get_url("http://www.apple.com")

on get_url(this_url)
set browser_string to "'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/125.4 (KHTML, like Gecko) Safari/125.9'"
return do shell script "curl " & (quoted form of this_url) & " -A " & browser_string
end get_url

When I run it, I get this error (which makes no sense to me):

curl: (7) Failed to connect to www.apple.com IP number 1: 61

From the curl man pages, I see that error 61 is "Unrecognized transfer encoding", but I don't know what that means.

Does anyone have a clue: a) why it doesn't work for me, and b) how to go about fixing it?

This is just a guess, because I'm not at my Mac and can't try it, but shouldn't the line:
return do shell script "curl " & (quoted form of this_url) & " -A " & browser_string
read:
return do shell script "curl " & this_url & " -A " & browser_string
?

1) Your AppleScript works for me on 10.3.8

2) When troubleshooting an AppleScript that uses 'do script' to run a shell command, you should always try running the shell command by itself in its native environment: the Terminal.
You could replace your 'do script' line with a line that set the script into a variable. Then run the AppleScript from Script Editor and it will print out what the value of that variable is. Copy & paste it into a Terminal window.

When I run it, the shell command is:
curl 'http://www.apple.com' -A 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/125.4 (KHTML, like Gecko) Safari/125.9'

Note that when writing a script, it is generally a good idea to use the full path to commands - so that you don't accidentally get some other version. I.e. use '/usr/bin/curl' instead of just 'curl'

That makes three (known) people for whom it works.
Running it it the terminal, either as curl etc, or as /usr/bin/curl etc. get exactly the same error statement.

Setting the shell script to an AppleScript variable just executes it and returns the usual error.

Clearly, my system is screwed up, I guess, and I don't know enough to test curl in any other way.

Thanks

1) Do you get this same problem if you use some other web site than Apple's?

2) Do you get this same problem if you omit the browser agent stuff (just do 'curl www.apple.com') ?

3) Do you have any proxies set up via the 'http_proxy' environment variable? Try 'printenv http_proxy'

4) What do you get if you do 'ping www.apple.com' (in Terminal)?

yes
yes
no response (just a new prompt)
It took 40 successful pings for me to recall that I needed a Control-c to stop it. Average ping round trip - 161 ms - not outrageous

So if understand correctly, you have no problem in accessing www.apple.com in Safari (for example) but doing 'curl www.apple.com' in Terminal gives you the same error message you quoted earlier. Is that correct?
And what do you get from 'which curl' in Terminal?

No problems with Safari, Foxfire, or Camino accessing any URL, but 'curl www.apple.com' fails as usual above.
which curl = /usr/bin/curl
I really appreciate this, hayne.

Adam

And does this happen if you login as a different user?
I'm thinking the problem must be related to something you've configured in your shell startup files. I assume you haven't modified any of the files under /etc

Bingo! Although I'm a single user, I make it a habit to install a completely virgin user account on every OS X machine called "Ghost" and I never alter it. Didn't think to try it, so fat lot of good it does me. In Ghost, via the Terminal, curl 'www.apple.com' runs exactly as predicted.

I have never knowingly altered /etc (don't know how to) but it sounds as if something has.

No - if you haven't changed the files under /etc, then ignore that. I asked about that because such changes would affect all users.

So it seems that the problem is in something in your user's shell environment. Please show us the contents of your .profile and .bashrc files:

cat ~/.profile
cat ~/.bashrc

cat: /Users/xxxxx/.profile: No such file or directory
cat: /Users/xxxxx/.bashrc: No such file or directory

where I've replaced my short name with "xxxxx"

Okay, how about:

cat ~/.login
cat ~/.bash_login
cat ~/.bash_profile
echo $BASH_ENV

cat ~/.login
##
# DELUXE-USR-LOCAL-BIN-INSERT
# (do not remove this comment)
##
echo $PATH | grep -q -s "/usr/local/bin"
if $status == 1 then
set path = ( "/usr/local/bin" $path )
endif
----
cat ~/.bash_login
cat: /Users/xxxxx/.bash_login: No such file or directory
----
cat ~/.bash_profile
cat: /Users/xxxxx/.bash_profile: No such file or directory
----
echo $BASH_ENV
tcsh: BASH_ENV: Undefined variable.

With so many modern applications "phoning home" and probably using curl to do it, one of them must have done something, somewhere.

The contents of that .login file suggest two things:
1) It was edited by some program (hence the weird comment)
2) It seems to be using syntax appropriate for the tcsh shell, not the bash shell which is the default shell in Panther and which I had been assuming is your shell.

To check which shell your account uses, try this:
echo $SHELL
This shouldn't matter since AppleScript's 'do script' always uses the bash shell.

You might consider temporarily moving that ~/.login file elsewhere
(e.g.: mv ~/.login ~/.login.save )
and then logging out and back in again to see if it makes a difference.

1) Result: echo $shell --> /bin/tcsh
2) "You might consider temporarily moving that ~/.login file elsewhere
(e.g.: mv ~/.login ~/.login.save ) and then logging out and back in again to see if it makes a difference."
Did the mv, logged out and in, no difference; shell still tcsh, same error from curl 'www.apple.ca'.
3) Typed bash at the prompt, got the bash shell (although echo $shell returns a blank line, the Terminal window title changes to Terminal -- bash -- 80x24), tried the AppleScript again - same failure. Tried the curl portion in Terminal, same failure.
4) Terminal Prefs have the radio button "Execute the default login shell using /usr/bin/login" selected. cat ~/.login now says "No such file or directory (as it should after moving it, I suppose.) Should I move it back, i.e. mv ~/.login.save ~/.login ?

Okay, so your shell is tcsh. That's fine - it probably means your account is one that came over from Jaguar.

Yes, you might as well put the ~/.login file back in place.

I'm about out of ideas at the moment. Maybe I would see something if you provided me with ssh access to your machine.

In fact from OS X 10.1 to Jaguar to Panther. How would I go about permanently changing it to bash if that's what's expected? By the way, so did my Ghost account, and the curl command works there, so it makes the transition to bash as required.

Done

Can do if you explain how. Unfortunately, I'll be gone for the next 3-1/2 hours. Not back until 5 your time.

You don't need to change your account's shell.
AppleScript's 'do script' always uses the bash shell no matter what your account's shell is. That is why your 'Ghost' account has no trouble even though its shell is tcsh.

Well, in desperation, I looked at the source code for curl (http://www.opensource.apple.com/darw.../10.3/curl-15/) and searched for the words of that error message. That way I found out that the error is coming when the 'connect' system call is failing and that the '61' at the end of the message is actually the C-programming language 'errno' (from /user/include/sys/errno.h) and thus the '61' means (as given in the man page for 'connect'):
"[ECONNREFUSED] The attempt to connect was forcefully rejected."

Looking closer at the message from 'curl' above, I now see that it starts with "curl: (7)" and the '7' is the error number that is being referred to in the 'curl' man page. It says that '7' means "Failed to connect to host".

Okay, now that we've got that cleared up, we know more about where to look for the problem.

A "connection refused" usually means that there is a firewall blocking the connection. Is it possible that you have some sort of non-standard firewall-type utility that is active only for your main user account and not for your 'Ghost' account?

Another thing you can try (in Terminal) in order to rule out this being a problem specific to 'curl' is try connecting to port 80 using 'telnet'.
E.g.:
telnet www.apple.com 80

When I try this, it takes longer than it should (tens of seconds) but I do connect successfully - I see the following:
Trying 17.254.0.91...
Connected to www.apple.com.
Escape character is '^]'.

I get: But much more important, a program called Little Snitch asks for permission to connect! That and your Ghost hint are the clue: Little Snitch informs me when a program is "reaching out" and allows me to permit it once or forever, creating a filter if forever. When I tried the telnet above, it asked. When I try the curl it doesn't. When I turn Little Snitch off in its preference pane, curl works as it should. Strange but true.

Thanks, Hayne - you are astonishingly patient!