Automate FTP download
 

Ok so am wondering how one might go about automating FTP download on OS X. I want to give it the remote FTP directory and have it download all files & folders in that directory at a set time each day.

So obviously cron would be perfect for this, except I no nothing about writing cron scripts. I can write in PHP, but that seems overkill and an unnecessary waste of resources just to get some automated FTP action going on.

Any ideas, help, or articles on doing such a thing?? I use transmit when i manually FTP, so I would guess I could do something in AppleScript... but I know nothing about AppleScript.

Here's an example of an AppleScript script that uses Transmit to download a remote directory to a local directory. You need to enter your info in the first few lines. It would be advisable to add some error checking to the script.

There are various ways besides cron to run scripts on a schedule. iCal 1.5.2 can handle it, as can Script Timer. A search at MacUpdate, VersionTracker or MacShareware.net should produce a few more candidates.

-- Rob

brilliant... many thanks!!

PS. I have iCal 1.5.2 and do not see where you can schedule scripts to run... do you just setup a recurring event and put the script path in the URL field?

hmm my install of AppleScript seem to be messed up. I don't seem to be able to compile or record any scripts.

hmm no good... I am downloading daily website backups, so the files need to be overwritten. The script just ends when it find a file already downloaded.

Thanks for pointing me in the right direction. I did not realize Transmit was developed to take advantage of AppleScript. A quick trip to the Panic homepage found some example scripts, one of which is a synchronization script. I did a little tweaking to make some of the values easier to change. here is what I have and working great it seems. Plus the sync mode will save me some bandwidth on unchanged items.

I wondered if you might want a synch script and I'm glad that you found one. :)

To run a script with iCal, set up the repeating event and then, from the alarm menu in the info pane, use "Open file" and select the script file.

-- Rob

much shorter way
 

... doesn't depend on Apple's or 3rd party apps, and it runs 100% behind the scenes

script:save it somewhere memorableMake it executable:
chmod +x myscript


now, for cron:

assuming we want it to happen every day at 2:30 pm, we want our cron line to look like:

30 14 * * * /full/path/to/my/script

as you don't know about cron much, I will assume you don't have any jobs you will miss. The following line will erase your current cron file and replace it:
otherwise you have to learn the VI editor (man vi)

Re: much shorter way
 

curl is how I would do it too...nope, just set your EDITOR variable to your favorite editor and use crontab -e:If your favorite editor is bbedit you will need to use a shell script as mentioned in the 'bbedit' man page.

edit: but you really should learn vi :D

...export EDITOR=pico...

By jove, Holmes! I think he's on to something!

well the only thing with curl is that I do not know the file names... there are many files to download and the names are not always the same. Also I'm not sure if curl will download an entire directory and its subdirectories. That and not all of the backup files change every day, but some do, so reloading everything would be a huge waste of bandwidth (seeing as currently if I were to do that I'd be downloading ~1GB per day because there are daily, weekly and monthly backups).

So the transmit sync script I think works the best for me because each day I only need to download ~300MB because the weekly and monthly do not change each day and will only be downloaded on the days they do change (once per week for weekly and once per month for monthly).

Sure the curl is nice and runs in the background, but when I'm downloading entire server backups I don't need to be wasting bandwidth on unchanged files and transmit has a very nice sync feature that is apple scriptable.

Re: much shorter way
 

That's a really bad idea to give to people. The Apple maintenance scripts calls are located in the crontab and believe me, you would miss them if they are gone. System performance can degrade very badly if they do not run as intended.

You would better off to tell people to run `sudo pico /etc/crontab` and add the line to the end of the file.

When I said I didn't know much about cron, I didn't mean cron. Actually I said "cron scripts" and probably should have said "I don't know much about writing shell script" to make myself clear.

Re: Re: much shorter way
 

though it may be a bad idea in this context, his command merely replaces your user's crontab, as he alluded to above. the system crontab at /etc/crontab remains intact.

caveat: it's best to understand commands proffered from the helpful populace.

Thanks Merv - took the words right out of my mouth.

Dreamscape: you did give the impression in your first post that the terminal was semi-foreign land for you, so I graded my suggestion accordingly. I deliberately didn't mention the difference between the system crontab and the user's - I presumed your's was empty and gave a very easy way to add a single line to it.

If you want to continue with background jobs, you should consider creating an archive process on your server - make a script to make an archive tarball, zip it and put it somewhere known. Call this script from the user crontab on the server.
curl does have some very powerful globbing and series commands that may work for you. Applescript just seems like a lot of overkill for a process that really doesn't involve the GUI.

I think you may mistunderstand what I am saying. I am not downloading the entire contents of the server. It's a cPanel controlled server and I have daily backups setup and since we do not currently have another server to send them to, I am currently dropping them in /backups.

There are daily, weekly, and monthly backups. The backups script automatically tarballs & gzips all accounts on the system + all databases + alot of other things (like aliases, crons, configs, etc, etc).

Currently, the daily backups comprise ~300MB total (in about 50 files I would guess). However, the daily backups does not always run, as it will stop if the load on the CPU is too high, to keep the server from crashing. So everyday I do not need to waste 300MB of bandwidth downloading the same files I did the day before. Also when you count in the weekly backups and monthly backups that do not change every day, that would be a total of ~900MB of bandwidth a day, 600MB completely wasted.

So I want to sync with the backups so that they are only downloaded when the backups actually ran, and the weekly and monthly are not downloaded everyday either, but rather only when the weekly or monthly have run.

I have found a nice solution to this with Transmit and Applescript. Only thing I do not like about it though, is that when the Applescript is running, I cannot do anything else in Transmit. It won't even let me open a new window.

I suggest that you provide feedback to Panic about this issue. Maybe there's something that they can do to overcome the problem in a future release. They have always responded kindly to my feedback and some of my requests/reports have resulted in improvements. :)

-- Rob

The other thing you could do is mount the ftp directory:Or you can use the keychain to store the username/password, but might have some keychain timeout problems that way or if you are not logged in...

Then you can use any normal script to look at the files and decide what to copy....

edit: Added USER:PASS info, no need for keychain

The easiest way seems to be , use the Command K in the finder or select connect to server in the menu and enter the server name, ueser id etc if requried. Once connceted make an alias for that.So when you need to connect just click the alias and your remote site will be mounted.

Dears, I'm also new to apple script and I need to download and then delete specific files (*.log) to a web server - can you help me?

Well, there's at least half a dozen ways. Which one we choose depends largely on how your webserver does things. Do you have one single file in a clearly defined place, or a directory? Is it accessable through http, or only ftp? Are the filenames descriptive? Do we really have to delete files off the server, or will the server take care of that for us?

Post the answers along with anything else that looks useful.

Ok - sorry - I'll esplain.
I need to download the LOG file of a Windows IIS web server, they are on a single place (The directory name is /logs/), the file have different name (ex. -> ex090804) but they have the same extension (.log)
I have only FTP access with username and password.
I need absolutely to delete them after download (the server doesnt do this).
Thank you very much

You can try this... i found it a while back but never used. I think it will work for you.
-- begin user defined variables --
set myUserName to "username"
set myPass to "password"
set ftpServer to "ftp.server.com"
set initialPathOnFTPserver to "/" -- directory to download
set local_folder to "path/to/files/"
-- end user defined variables --

tell application "Transmit"
activate
set newWindow to make new document at end
tell newWindow
set your stuff to local_folder
connect to ftpServer as user myUserName with password myPass ¬
with initial path initialPathOnFTPserver with connection type FTP
download item initialPathOnFTPserver
disconnect
end tell
quit
end tell