shell script for ftp?

jbc · 06-07-2003, 01:55 PM

I just waded into the man pages for ftp for the first time. There's an overwhelming number of options, and I'm not that familiar with ftp. I have an ftp task that I would like to automate with a shell script. I'm hoping that someone can give me some clues on the minimum ftp options that need to be set, or tell me if this is even possible in a shell script.

Basically I need to upload the contents of a particular folder on my system to a web site on a Windows server via ftp Friday of every week. Logging into the ftp server with our username/password takes me directly to where I need to upload the files.

It looks as if I need to use something like mput to tell ftp to send every file in the folder (mput /Users/jbc/StuffToUp/* maybe?), but I don't know if this is an option that can only be used in interactive mode. Also I want to be sure not to send the .DS_Store file along with the pdf and text files in the folder.

As far as the other ftp options, I just don't know which ones I need to be concerned with for the upload to work correctly.

Simple task it seems like, but I have no idea where to start putting together a script to do this. I don't want to mess up the server with a trial and error approach. Again, some basic orientation would be much appreciated!

Brad

yellow · 06-07-2003, 02:02 PM

You might want to take a peek at the curl man page. It'll upload multiple files from a directory. Maybe it'd be easier use it in a script? (yellow != scripter)

jbc · 06-07-2003, 02:08 PM

Really! I thought curl was just for downloading files; shows you how little I know. I'll look into it. Thanks, yellow!

yellow · 06-07-2003, 02:15 PM

Same here.. someone enlightened me recently. I'd say primarily it's a downloader, but it does support uploading too. Long, boring man page.

honestpuck · 06-08-2003, 09:47 AM

Quote:


	Originally posted by jbc Basically I need to upload the contents of a particular folder on my system to a web site on a Windows server via ftp Friday of every week. Logging into the ftp server with our username/password takes me directly to where I need to upload the files. Brad

You might want to also look at rsync. It's really magic and a lot faster as it only transfers files that have changed.

Tony

macmath · 06-08-2003, 10:32 AM

You might look at these hints from the front page:

One for rsync

This one and this one for ftp transfers.

jbc · 06-08-2003, 12:08 PM

Hmm. I'm beginning to see the crux of the problem.

The second ftp link of macmath's post gives a basic script for uploading a file (thank you!). Of two comments on the hint, one asks how to use msend (= mput) to upload an entire directory; no reponse to the question in over a year.

The suggested rsync is obviously great for tranferring whole directories, but doesn't seem to have any facilities for working over an ftp connection (uses it's own protocol as near as I can tell; not an option in this case, since remote Windows system is set up to do this via ftp).

So looks like I need to do some research on name-globbing and mput/msend options for ftp command.

mervTormel · 06-08-2003, 12:51 PM

it's pretty easy to script once you've done the deed manually:

Code:

$ pwd
/Users/merv/work

$ ls pathb/
foo1.txt  foo2.txt  foo3.txt  foo4.txt

$ ftp ftp.mindspring.com
Trying 199.174.114.51...
Connected to www.mindspring.com.
220 FTP server ready
331 Password required for user
230-    Your disk quota is: 10.00 megabytes
230-    Your disk usage is: 3.30 megabytes (33.03% of quota)
230-
230 User  logged in.
Remote system type is UNIX.
Using binary mode to transfer files.

ftp> lcd pathb
Local directory now /Users/merv/work/pathb

ftp> mput *.txt
mput foo1.txt [anpqy?]? q
mput aborted.

ftp> prompt
Interactive mode off.

ftp> mput *.txt

local: foo1.txt remote: foo1.txt
500 EPSV not understood.
227 Entering Passive Mode (199,174,114,51,6,83).
150 Opening BINARY mode data connection for foo1.txt
226 Transfer complete.
local: foo2.txt remote: foo2.txt
227 Entering Passive Mode (199,174,114,51,6,84).
150 Opening BINARY mode data connection for foo2.txt
226 Transfer complete.
local: foo3.txt remote: foo3.txt
227 Entering Passive Mode (199,174,114,51,6,85).
150 Opening BINARY mode data connection for foo3.txt
226 Transfer complete.
local: foo4.txt remote: foo4.txt
227 Entering Passive Mode (199,174,114,51,6,87).
150 Opening BINARY mode data connection for foo4.txt
226 Transfer complete.

macmath · 06-08-2003, 02:06 PM

I'm sorry about the rsync link. I did not read it carefully enough...I found it and the others by doing a search on 'script AND ftp' and I just assumed that it was ftp related.

About the other item. I use lftp, and using mput is just a matter of saying 'mput *.*' or 'mput *' [or mput path/to/file/*.* or mput path/to/file/* if you have not placed lftp in the appropriate directory (edited to add this)]. I have not tried that with the installed ftp, but checking the man pages for ftp ought to clear that up. You could use fink to install lftp, or google lftp, download it and install it yourself (if you have the developer tools installed).

macmath · 06-08-2003, 02:24 PM

I use the following script to gzip the files I've placed in a 'transfer' directory over the course of the day so that they can be uploaded to another site by ftp (and then I download them from that site when I get home (and vice-versa from home to work). This script is called by cron at the end of the work day.

It gzips the files, connects to the site, changes directory, uploads to that directory, then deletes the contents of the 'transfer' directory.
-------
#! /bin/tsch
echo Moving to directory...
cd /Users/us/Documents/Ed/Transfer/
echo gzipping items...
gzip /Users/us/Documents/Ed/Transfer/*
echo Uploading items...
lftp <<**
open address.of.ftpsite
cd intransit
mput /Users/us/documents/ed/Transfer/*.gz
bye
**
echo FTP ended - the items have been uploaded
echo deleting items
mv /Users/us/Documents/Ed/Transfer/* /Users/us/.Trash
echo FTP ended - the item have been deleted
-------

Of course, you'll need a .netrc file to make this work (properly protected from prying eyes).

This is (approximately) my 100pi'th post.

jbc · 06-08-2003, 03:59 PM

Quote:


	Originally posted by macmath This is (approximately) my 100pi'th post.

Hehe. All you need is to figure out how to make 0.159... posts and you've got it!

Thanks merv and macmath. Trying to clean up the Applescript that generates the files to upload, then I'll try putting the pieces together to ftp them.

Btw, does the name string for mput support "|" as an OR operator? In my case, can I do something like

Code:

mput /Users/Shared/FilesToUp/*(.txt|.pdf)

or am I getting this mixed up with regular expressions?

mervTormel · 06-08-2003, 04:15 PM

you're mixed up.

globbing is done as in csh, according to the ftp man page.

this works:

Code:

$ ls *{.txt,.pdf}
foo1.pdf  foo1.txt  foo2.pdf  foo2.txt  foo3.pdf  foo3.txt  foo4.pdf  foo4.txt

...

ftp> mput /path/to/*{.txt,.pdf}

mervTormel · 06-08-2003, 04:40 PM

jbc, seems like a good example can relieve some pain. be sure to read the ftp man page about the .netrc file and your shell man page about 'here documents':

Code:

$ cat ~/.netrc:

machine ftp.domain.com login username password "s3cr3t"


basic shell script to upload multiple files to some ftp space:

#!/bin/sh

/usr/bin/ftp ftp.domain.com <<__end

prompt off

cd /path/to/targetDir

mput /path/to/sourceDir/*{.txt,.pdf}

quit

__end

jbc · 06-08-2003, 09:33 PM

Okay, got it to work! Had to adopt merv's lcd approach from the interactive version above to get it to go, otherwise the remote file was specified as having a path just like the local file, which caused the the ftp server to deny access 218 times on the first attempt. But after that, running the script from the Script menu worked perfectly.

I'm surprised it was so simple - basically merv's last example with the addition of a line to change the local working directory; I would have thought more options needed to be specified to get the ftp command talking to another computer. I'm really beginning to like unix!

Not used to seeing the ftp info scolling by in MkConsole. Looks like this for 218 files.

Code:

227 Entering Passive Mode (66,27,53,187,15,96).
125 Data connection already open; Transfer starting.
226 Transfer complete.
49950 bytes sent in 00:16 (2.87 KB/s)
local: 0322NV016SparksR.pdf remote: 0322NV016SparksR.pdf
227 Entering Passive Mode (66,27,53,187,15,103).
125 Data connection already open; Transfer starting.
226 Transfer complete.
39678 bytes sent in 00:16 (2.36 KB/s)
local: 0322NV016SparksT.pdf remote: 0322NV016SparksT.pdf

Is it normal to see "Entering Passive Mode" and "Data connection already open" for every file? Seems redundant somehow.

Thank you merv and macmath for walking me through this! Much appreciated.

Brad

mervTormel · 06-08-2003, 09:51 PM

yeah, ftp client / server conversations are noisy like that because clients and servers have unknown capabilities and chattiness and clients just try something and the server says something chatty back.

it's like an angry, disagreeable phone conversation that gets the job done with a lot of "no way!" and "i can't believe you don't offer that, stupido!" in between.

06-07-2003, 01:55 PM	#1
jbc All Star Join Date: Feb 2003 Location: Chico, CA Posts: 686	shell script for ftp? I just waded into the man pages for ftp for the first time. There's an overwhelming number of options, and I'm not that familiar with ftp. I have an ftp task that I would like to automate with a shell script. I'm hoping that someone can give me some clues on the minimum ftp options that need to be set, or tell me if this is even possible in a shell script. Basically I need to upload the contents of a particular folder on my system to a web site on a Windows server via ftp Friday of every week. Logging into the ftp server with our username/password takes me directly to where I need to upload the files. It looks as if I need to use something like mput to tell ftp to send every file in the folder (mput /Users/jbc/StuffToUp/* maybe?), but I don't know if this is an option that can only be used in interactive mode. Also I want to be sure not to send the .DS_Store file along with the pdf and text files in the folder. As far as the other ftp options, I just don't know which ones I need to be concerned with for the upload to work correctly. Simple task it seems like, but I have no idea where to start putting together a script to do this. I don't want to mess up the server with a trial and error approach. Again, some basic orientation would be much appreciated! Brad

06-08-2003, 10:32 AM	#6
macmath MVP Join Date: Mar 2002 Location: Elsewhere Posts: 1,497	From the Hints page: You might look at these hints from the front page: One for rsync This one and this one for ftp transfers.

06-08-2003, 02:06 PM	#9
macmath MVP Join Date: Mar 2002 Location: Elsewhere Posts: 1,497	I'm sorry about the rsync link. I did not read it carefully enough...I found it and the others by doing a search on 'script AND ftp' and I just assumed that it was ftp related. About the other item. I use lftp, and using mput is just a matter of saying 'mput .' or 'mput ' [or mput path/to/file/.* or mput path/to/file/* if you have not placed lftp in the appropriate directory (edited to add this)]. I have not tried that with the installed ftp, but checking the man pages for ftp ought to clear that up. You could use fink to install lftp, or google lftp, download it and install it yourself (if you have the developer tools installed). Last edited by macmath; 06-08-2003 at 02:18 PM.

06-08-2003, 04:15 PM	#12
mervTormel League Commissioner Join Date: Jan 2002 Posts: 5,536	you're mixed up. globbing is done as in csh, according to the ftp man page. this works: Code: $ ls {.txt,.pdf} foo1.pdf foo1.txt foo2.pdf foo2.txt foo3.pdf foo3.txt foo4.pdf foo4.txt ... ftp> mput /path/to/{.txt,.pdf} __________________ On a clear disk, you can seek forever.

06-08-2003, 04:40 PM	#13
mervTormel League Commissioner Join Date: Jan 2002 Posts: 5,536	jbc, seems like a good example can relieve some pain. be sure to read the ftp man page about the .netrc file and your shell man page about 'here documents': Code: $ cat ~/.netrc: machine ftp.domain.com login username password "s3cr3t" basic shell script to upload multiple files to some ftp space: #!/bin/sh /usr/bin/ftp ftp.domain.com <<__end prompt off cd /path/to/targetDir mput /path/to/sourceDir/*{.txt,.pdf} quit __end __________________ On a clear disk, you can seek forever.

06-07-2003, 02:02 PM	#2
yellow Moderator Join Date: Jan 2002 Posts: 10,677	You might want to take a peek at the curl man page. It'll upload multiple files from a directory. Maybe it'd be easier use it in a script? (yellow != scripter)

06-07-2003, 02:08 PM	#3
jbc All Star Join Date: Feb 2003 Location: Chico, CA Posts: 686	Really! I thought curl was just for downloading files; shows you how little I know. I'll look into it. Thanks, yellow!

06-07-2003, 02:15 PM	#4
yellow Moderator Join Date: Jan 2002 Posts: 10,677	Same here.. someone enlightened me recently. I'd say primarily it's a downloader, but it does support uploading too. Long, boring man page.

06-08-2003, 12:08 PM	#7
jbc All Star Join Date: Feb 2003 Location: Chico, CA Posts: 686	Hmm. I'm beginning to see the crux of the problem. The second ftp link of macmath's post gives a basic script for uploading a file (thank you!). Of two comments on the hint, one asks how to use msend (= mput) to upload an entire directory; no reponse to the question in over a year. The suggested rsync is obviously great for tranferring whole directories, but doesn't seem to have any facilities for working over an ftp connection (uses it's own protocol as near as I can tell; not an option in this case, since remote Windows system is set up to do this via ftp). So looks like I need to do some research on name-globbing and mput/msend options for ftp command.

06-08-2003, 02:24 PM	#10
macmath MVP Join Date: Mar 2002 Location: Elsewhere Posts: 1,497	I use the following script to gzip the files I've placed in a 'transfer' directory over the course of the day so that they can be uploaded to another site by ftp (and then I download them from that site when I get home (and vice-versa from home to work). This script is called by cron at the end of the work day. It gzips the files, connects to the site, changes directory, uploads to that directory, then deletes the contents of the 'transfer' directory. ------- #! /bin/tsch echo Moving to directory... cd /Users/us/Documents/Ed/Transfer/ echo gzipping items... gzip /Users/us/Documents/Ed/Transfer/* echo Uploading items... lftp <<** open address.of.ftpsite cd intransit mput /Users/us/documents/ed/Transfer/.gz bye * echo FTP ended - the items have been uploaded echo deleting items mv /Users/us/Documents/Ed/Transfer/* /Users/us/.Trash echo FTP ended - the item have been deleted ------- Of course, you'll need a .netrc file to make this work (properly protected from prying eyes). This is (approximately) my 100pi'th post.

06-08-2003, 09:33 PM	#14
jbc All Star Join Date: Feb 2003 Location: Chico, CA Posts: 686	Okay, got it to work! Had to adopt merv's lcd approach from the interactive version above to get it to go, otherwise the remote file was specified as having a path just like the local file, which caused the the ftp server to deny access 218 times on the first attempt. But after that, running the script from the Script menu worked perfectly. I'm surprised it was so simple - basically merv's last example with the addition of a line to change the local working directory; I would have thought more options needed to be specified to get the ftp command talking to another computer. I'm really beginning to like unix! Not used to seeing the ftp info scolling by in MkConsole. Looks like this for 218 files. Code: 227 Entering Passive Mode (66,27,53,187,15,96). 125 Data connection already open; Transfer starting. 226 Transfer complete. 49950 bytes sent in 00:16 (2.87 KB/s) local: 0322NV016SparksR.pdf remote: 0322NV016SparksR.pdf 227 Entering Passive Mode (66,27,53,187,15,103). 125 Data connection already open; Transfer starting. 226 Transfer complete. 39678 bytes sent in 00:16 (2.36 KB/s) local: 0322NV016SparksT.pdf remote: 0322NV016SparksT.pdf Is it normal to see "Entering Passive Mode" and "Data connection already open" for every file? Seems redundant somehow. Thank you merv and macmath for walking me through this! Much appreciated. Brad

06-08-2003, 09:51 PM	#15
mervTormel League Commissioner Join Date: Jan 2002 Posts: 5,536	yeah, ftp client / server conversations are noisy like that because clients and servers have unknown capabilities and chattiness and clients just try something and the server says something chatty back. it's like an angry, disagreeable phone conversation that gets the job done with a lot of "no way!" and "i can't believe you don't offer that, stupido!" in between. __________________ On a clear disk, you can seek forever.