What are the advantages of using the Terminal UNIX commands?
 

I am only aware of UNIX from my dealings with OSX, but I was a 'power' user of OS9.

I understand the power of the 'stability' of the UNIX underpinnings within OSX but what I am really interested in is how can understanding UNIX make me more productive within OSX?

I understand it is possible to have 'auto run' scripts and files that will automatically back up stuff for me.... etc. etc.

So anyone got any insights as to how they use UNIX commands in a productive way? Rather than simply repeating what I can already do using a GUI?

I am genuinely interested to know what 'extras' UNIX can give me.

all,

unix command line is all about sheer, raw, unadulterated power.

the philosophy of unix is to create small tools that do little things very well and with many options. and to be able to chain them together into larger tools.

there's a lot to learn. unix command line can constitute quite a commitment.

there are jobs to be done that just can't be accomplished adequately with a GUI app, no matter how great the programmer.

e.g. (perhaps contrived, but it drove me to learn)

i wanted to collect all my surf rock mp3 files (260 of 'em) into a flat directory so i could burn them to CD and play them in my vehicle's mp3 player, which requires a flat directory structure.

unfortunatley, all my mp3's are categorized by genre/artist/album so with a GUI app, even MP3 Rage, it would be very tedious and take much mousing.

so, how to accomplish this?

here's _one_ way (possibly tedious to learn, for some, but i found it irksome and joyful at the same time) ...

% find /Volumes/music/SurfRock/ -type f | sed 's/^/"/;s/$/"/' | xargs -i \
cp --backup=numbered --preserve {} /Volumes/scratch/surfnturf/


all the machinations are necessary because some of the files contain spaces and single quotes and other special characters that need to become benign in the full path reference to the file.

let's look at what this does (fairly well)

- finds the mp3 files which are type f (plain old files to unix)

- sed puts quotes at the beginning and end of the file reference

- pipe that to xargs which executes the cp (copy) command on the file

- cp the file to target dir, number duplicate files (several versions of Gringo and Surf Rider), preserve owner-date-permission meta-data

[ N.B. i am using the fink toolset, so some commands, like cp, are the GNU versions with their switches, like --backup=opt ]

so, if you carry out the logical conclusion of this, you will see that there are always some special jobs to be done that are terribly cumbersome in GUI apps that are not designed to do them, but are handled quite elegantly (if not rococo) by the power of the thousands of command line tools available in unix.

but i wax... hope that helps. keep digging.

-b

Hi -b

Now that is the sort of thing I'm looking for exactly. Thanks for taking the time to explain this.

I think I got it all, except for...

Which bit is the pipe? Is that the 's/^/"/;s/$/"/' ???? If so what is that doing exactly?

But again, I understood this is the sort of thing that UNIX was good at, but the example is excellent.

Another question... I like this, and see the great potential, but having to type that line in everytime one wants to run that has to be a little long winded. Is it possible to save such lines in some way for later recall?

Nice example ... here's another
 

The main macosxhints' website, including both the database (the stories and users) and the site content (HTML files and images) are backed up to my local Mac via the UNIX environment. The stories are backed up twice a day, and the site content is backed up once a week.

How is this done? It's actually relatively simple when it's two UNIX hosts speaking to each other. As an example, here's the "database" backup script. Don't let it intimidate you; I'll explain each line (with a "#" below the main line as a comment indicator):

That's all there is to it. I then used "cron", the UNIX scheduler, to set this to run twice a day. My backups now happen automatically without interruption, as long as the home machine is powered on (and not sleeping).

The remote script that's called on the webserver is very simple. mySQL has a "dump" command which basically outputs the whole database structure and contents in one text file, suitable for importing into another database. So the remote script just issues that command on the macosxhints' database.

I also have a cron job that looks in the backup directory on a regular basis and deletes any backups older than three days (for the database) or three weeks (for the site).

-rob.

The pipe is "|" (vertical bar) character. It causes the output of the command on the left to feed-in or pipe into the input of the command to the right.

E.g.

ls -l | more

ls' "long" output is piped into the more command which displays one page at a time until you press the space bar to go to the next page.

hey chadders,

i'm tingly that you appreciate my post.

here's where the commitment of which i spake comes in.


the pipe, the vertical bar character '|', is the magical character that chains the output of one command to the input of another.

so, for example, if you run the commands one at a time, you will see the net effect. i'll run each command on my system and show you excerpts of their results...

% find /Volumes/music/SurfRock/ -type f | sed 's/^/"/;s/$/"/' | xargs -i \
cp --backup=numbered --preserve {} /Volumes/scratch/surfnturf/

first part:

what does

% find /path -type f

get you?...

% find /Volumes/SurfRock/ -type f | more
/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Last Date
/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Sleepwalk
/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Driving Guitars
/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/I'm Branded
/Volumes/SurfRock/Los Straitjackets/Delphonic Sounds Today/Surf Rider
...

okay, see the spaces and the single quote character in "I'm Branded" ?

those are going to choke the xargs cp command, so we have to munge the data a little to make it play nice.

we need to enclose the output of the find command with double-quotes to make them simple strings to the xargs command.

sed, the stream editor, does this neatly; so what does

sed 's/^/"/;s/$/"/'

do?...


it says "substitute the beginning and end of the line fed me with a double-quote."

so adding that as a piped command you get


% find /Volumes/music/SurfRock/ -type f | sed 's/^/"/;s/$/"/'

% find /Volumes/music/SurfRock/ -type f | sed 's/^/"/;s/$/"/'
"/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Last Date"
"/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Sleepwalk"
"/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Driving Guitars"
"/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/I'm Branded"
"/Volumes/SurfRock/Los Straitjackets/Delphonic Sounds Today/Surf Rider"

it merely wraps the output lines of the find command with double-quotes.

now, we can pipe this output to the xargs command and it won't choke on spaces and other special characters.

% find /Volumes/music/SurfRock/ -type f | sed 's/^/"/;s/$/"/' | xargs -i \
cp --backup=numbered --preserve {} /Volumes/scratch/surfnturf/


xargs simply says "execute the specified command on the arguments (input) from the pipe...

so, effectively, the xargs command execututes copy like so...

% cp --options {source} /target


xargs -i stipulates that the input from the pipe will be represented in the command as {} - open-close curly braces.

results:

% find /Volumes/music/SurfRock/ -type f | sed 's/^/"/;s/$/"/' | xargs -i \
cp -v {} /Volumes/flivver/surfnturf/

`/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Last Date' -> `/Volumes/flivver/surfnturf/Last Date'
`/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Sleepwalk' -> `/Volumes/flivver/surfnturf/Sleepwalk'
`/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/Driving Guitars' -> `/Volumes/flivver/surfnturf/Driving Guitars'
`/Volumes/SurfRock/Los Straitjackets/Damas Y Caballeros/I\'m Branded' -> `/Volumes/flivver/surfnturf/I\'m Branded'
`/Volumes/SurfRock/Los Straitjackets/Delphonic Sounds Today/Surf Rider' -> `/Volumes/flivver/surfnturf/Surf Rider'

Re: Nice example ... here's another
 

Wow this stuff is amazing... ok, I understand these lines, I think and can see what is happening but when you say this is a 'shell script' what do you mean by that? I assume these commands are all put together in some sort of 'script' that you can execute with a single command. how does one creat a script? Test a script to ensure it all works first?

And that cron job sounds interesting... that is a UNIX command that runs the script at set times? Yes?

I have learn't more in the last few hours about UNIX than I have so far reading 'UNIX Complete' which is great at telling me what all the commands do, and what they have as options, but I see the real power with UNIX is how those commands are strung together.

Can anyone recommend any books that show how to 'think' about using UNIX, rather than what they do?

Glad to be of service! :-)

Also thanks for the explanation here too, I now understant the pipe command much better. I did not realise that you could pipe the output of one caommand into another like that.

This is a whole new world opening up to me, I can see.

That's one of the things I love about OSX, it enables me to get things done in a productive way by being beautifully simple at one level and yet when I want to explore deeper there is so much more there for me!

I'm sure I will love the journey... can you recommend any useful books for a UNIX Newbie? It seems to me that much of the skill is in 'thinking' right about what you want to do, and then finding the UNIX commands to help you do it. Correct?

shell scripts
 

what is a shell?

from the Bash reference manual:

http://www.gnu.org/manual/bash-2.02/...href.html#SEC3

At its base, a shell is simply a macro processor that executes commands. A Unix shell is both a command interpreter, which provides the user interface to the rich set of Unix utilities, and a programming language, allowing these utilitites to be combined. Files containing commands can be created, and become commands themselves. These new commands have the same status as system commands in directories like `/bin', allowing users or groups to establish custom environments.


what is a shell script?

it is a collection of one or more commands to execute, saved in a text file, designated as executable, and fed to the shell as a command to run



books:

unfortunately, the authorative manual for unix is the man pages, which is largely esoteric and inconsistent.

% man tcsh
% man find
% man sed
% man xargs
% man bash

after that, there are some top notch books that deal with unices, but it's largely dependent on which way you want to go. beware, some are dreadly awful. research is necessary.


dig in

--

if it was easy, we wouldn't call it code.

my humble perspective
 

I'm also coming from an OS9 competent perspective. There are three main reasons I'm interested in the UNIX "underpinnings":

The first is scripting. Applescripts from OS9 are now joined by shell scripts and Perl scripts. With cron you've got the option of building hundreds of tiny robots to do work on your computer for you 24 hours a day. Assuming you've got anything repetitive to do, anyway.

The second is Open Source Software and Unix freeware. Apache, bind, sendmail, PHP, perl, MySQL, etc are there waiting to turn my humble computer into a full fledged no-holds barred web god. The equivelent of thousands of dollars of OS9 software are usable. Assuming you spend the hours to learn them, anyway.

The third is that I'm curious. I poked at OS9 like I'm poking at OS X. I wrote scripts for stuff, played with the extensions folder. Installed stuff just to see what it was like. Uninstalled other stuff, just to see if it would go faster. OSX dropped a hundred thousand little icon-less files onto my computer. I'm psychologically unable to just let them sit there.

Anyway, I just wanted to chime in with a little portrayal of the 'big picture' as I see it. Hope it helps in some way.

s

good input
 

sssss,

thanks for mentioning that stuff. all very good points. be sure to read griffman's '20 minute story' over in OS Xperiences in the General Discussions.

as for scripts, testing is very very important, as the iTunes 2.0.1 debacle bore out.

it can't be stressed enough to make small changes and test and test again. and build in error checking. and test return values.

another lesson i have learned:

if it worked yesterday, and nothing changed, and root can run it, it's probably a permission problem (lo, something did change!)

"Can anyone recommend any books that show how to 'think' about using UNIX, rather than what they do?"

The classic in the field is probably <UNIX Power Tools>, published by O'Reilly.

It's pricey, but look at that Table of Contents and you'll see why. :D

Books...
 

The first UNIX book I read was Sam's "Teach yourself UNIX in 24 hrs". It assumes zero knowledge of UNIX, and explains everything in fairly good detail.

Not perfect, but it does explain the "why" along with the "how".

-rob.

Lesson number one!
 

Thanks for the book pointers guys, I'm off to study amazon's collection...

but before I do... here is a starter for ten!

I want to use UNIX to help with my backup strategy...so...

Assume I have 3 partitions set up on one Drive. I wish to copy several files/directories (always the same files and directories) from different locations on the three partitions.

I then wish to copy these files to another location on another drive (always the same location) and I want to run that procedure every day at 10:00am.

I would also like a report producing of what happened, so that I can check that everything ran as expected.

Is this possible using a script and cron? If so anyone interested in having a stab at talking me through the process. I'll try to figure it out here myself as we go along if any are prepared to talk me through the thinking.

Thanks in anticipation! :-)

Re: Lesson number one!
 

yep. it's possible, and very doable.

here's a great web page which discusses it in detail...

http://www.bombich.com/mactips/image.html


here's my ditto script which will create a bootable Mac OS X volume

-----

#! /bin/sh

# dittoBackup.sh - ditto copy system volume to target

# dittp copy visible folders:

sudo ditto -rsrc /Applications /Volumes/whiskey/Applications
sudo ditto -rsrc /Developer /Volumes/whiskey/Developer
sudo ditto -rsrc /Library /Volumes/whiskey/Library
sudo ditto -rsrc /Network /Volumes/whiskey/Network

#sudo ditto -rsrc /sw /Volumes/whiskey/sw
# /sw permantly located off root and symlinked

sudo ditto -rsrc /System /Volumes/whiskey/System
sudo ditto -rsrc /Users /Volumes/whiskey/Users

# Next, use ditto to copy your Darwin system files:

sudo ditto -rsrc /private /Volumes/whiskey/private
sudo ditto -rsrc /usr /Volumes/whiskey/usr
sudo ditto -rsrc /bin /Volumes/whiskey/bin
sudo ditto -rsrc /sbin /Volumes/whiskey/sbin
sudo ditto -rsrc /mach_kernel /Volumes/whiskey/mach_kernel
sudo ditto -rsrc /.hidden /Volumes/whiskey/.hidden

# Finally, recreate symbolic links and empty directories:

cd /Volumes/whiskey
ln -s private/etc etc
ln -s private/var var
ln -s private/cores cores
ln -s private/tmp tmp

# add link for /sw
ln -s /Volumes/zulu/sw /Volumes/whiskey/sw

mkdir dev Volumes Network

exit

-----


this can be enhanced with echos to logfiles and hooked up to cron

i leave that as an exercise for the student at this juncture

get this tested and working to your liking, then we can explore cron'ning it

--
Wolverhampton Wanderers, who beat Lester, 10-nil

Re: Re: Lesson number one!
 

OK, I think I got this... and thanks for the link.

First I had to look up the ditto command, but I have a reference manual for UNIX here, so I understand that now seems logical(ish).

Why the need to create symbolic links?

And I am not sure what these lines are doing:

And I will certainly take a look at cron to be able to automate it.

Thanks for taking the time to post, it is appreciated.

hi chad,

ditto is prolly unique to OSX. it copies resource forks and maintains the inode meta-data.

why need to create the symlinks?

because they're (possibly) not there. think about the target of this backup as a freshly inited drive. and we're selectively copying from the source, but not _all_ from the source, just enough to make a bootable image of the source.

therefore, we:

- copy the directories of concern

but what about the symlinks that make a bonafide OSX?

- we re-create the symbolic links that are needed to be a true bootable OSX, but we need to translate those links to point to the intended targets on the backup volume

# = line commented out, skipped by the shell, in this case, old code

that ditto /sw is commented out because i moved /sw permanently off of the root dir / and i don't need to ever back it up since i can just update it from fink

i comment out changes to reflect that it was changed. a built in history of activity, so to speak

/sw is the fink port package installation. worry about that later.

regrets if it confused you.

merv,

I must admit, I was a little confused, but that's not hard I assure you... but your explanantion clears it up nicely.

Thanks for taking the time.

Cheers.

"thinking unix"
 

Chadders
take a look at:
"Think Unix"
by Jon Lasser
this will introduce you to the design/philosophical underpinnings of Unix as well as the Unix approach to computing.
I really think it should be required reading for unix newbies

Re: Re: Nice example ... here's another
 

Nice question. I like your emphasis on 'think'. The best work on how to think 'Unix' is "The Unix Programming Environment" by Kernighan and Pike. It is so far better than anything else that it remains a classic despite being written over 20 years ago.

After UPE almost anything will do you but I agree with those that recommend "Unix Power Tools".

Tony