Updating locate.updatedb daily instead of weekly
 

I want my locate database to update each day instead of just once a week. From what I can find, the database is updated weekly because of the following code in the /etc/weekly file:
In order to update the database on a daily basis, my first thought is that I need to move this code from /etc/weekly to /etc/daily. The only line I can see that I should edit would be to change the echo command in line 4 to say "daily" instead of "weekly".

Also, looking at the weekly script, it appears the PATH and host statements are not repeated so I think I'd need to keep that in there so the remaining part of the script will run properly. That would make the top part of my weekly script look like this:
Before I do this, I thought it best that I ask to be sure I'm making the correct changes. It's far easier to "not do" than to "un do". :)

So... am I even close to figuring this out correctly?

[edit: readability -mt]

if you look to the bottom of /etc/daily you'll see a callout to /etc/daily.local
what you really want to do in this case is:

a) avoid the mess of modifying standard system scripts because those changes have to be managed with updates. retro-fitting mods is a tiresome, tedious chore.

b) clone the locate.updatedb code into /etc/daily.local

as for the details, you should make a test script in your ~/bin and test it really well there ( a la sudo ) before deploying it to production /etc/daily.local

Macjanitor
 

Why not use Macjanitor for this?

http://personalpages.tds.net/~brian_...acjanitor.html

You can run the task anytime you feel like, but if you want to have fun in Terminal and find your own way it would be a good learning experience. I tried to do what you're thinking and I found I have to mess with "cron/crontab", not recommended for the faint at heart. However I've played with some tasks utilised Cronnix to "play" with them:

http://www.koch-schmidt.de/cronnix/


Good luck

MacJanitor is just a GUI wrapper for the following commands:
sudo sh /etc/daily
sudo sh /etc/weekly
sudo sh /etc/monthly

So you could easily make an alias to those commands or just plop it into your crontab.

But I think the daily.local is a much more elegant solution, which stays more in line with the way the system is set up to run anyway.

How simple! Done. Tested. Success. Thank you! :D

Yes, that's why I don't want to use MacJanitor. The more I learn how to use UNIX and the Terminal, the more flexibility I will have in the long run. While MacJanitor might have been great for this sort of thing (don't know, don't use it), it limits me to whatever the programmer thought I wanted to do. What if I want to do something different? I'd be dependent on hoping some other programmer came up with something that would do what I want. Instead, I want the freedom to do whatever I want, whenever I want to do it. I have the tools I need right now. I just have to learn how to use them. And so that's what I'm trying to do. Not that there's anything wrong with using a GUI app. I just want to do it myself.

I have no problems with GUI apps in general -- I mean, I use a GUI interface every day instead of running everything from the command line.

But sometimes I feel that people get too dependent on a GUI app which "simplifies" an already simple command.

The worse scenario is when that GUI app does something that I can do from the command line easily, and then the developer charges money for the GUI wrapper. (I know MacJanitor doesn't fall into this category)

Re: Updating locate.updatedb daily instead of weekly
 

I have made another change to this Vicky, I changed the above bit to:
The 'su -fm nobody' in the original limits the search to publicly searchable directories, where I want my whole machine searched, and since I am not up at whatever time it runs, no need to be 'nice' about it! 8-) .

[edit: readability -mt]

thanks, doug, good tip.

here's a dilemma. with the above config, the daily runs the locate update mechanism, but that one day when the weekly runs, it runs after the daily and stomps the work the daily has done, right?

so, how can we efficiently tell the weekly code to skip the locate update?

i think it'll take a little mod to the weekly script.

I can pretty much follow Doug's suggestion (in concept, totally, but reading the code is another matter). I'm trying to figure out what | nice -5 su -fm nobody 2>&1 | \ sed -e means.

nice -5: modifies a command to alter it's scheduling priority. man doesn't explain what the incriments represent so I don't know what -5 means.

su -fm: superuser. don't read the .cshrc file. don't alter environment.

2>&1: redirect output to a file? I'm guessing here. could &1 refer to the var, locdb, or to the file itself, locate.updatedb? Obviously, I don't know what & means. And why 2? Why 1?

sed -e: this is what creates the database, right? it reads the specified files.

And I don't know what the | and \ refers to, but it's obviously key.

UGH! I hate not being able to figure this stuff out yet.

Can someone read this line to me in English? I know this is over my newbie head, but I would like to understand it, and I'm not that far along in my books yet. The reason I'm interested in understanding this line is because I can't figure out how removing | nice -5 su -fm nobody 2>&1 (as I understood Doug to suggest) would make the script search the entire HD and not just the publicly searchable files. I can't figure out anything that controls which files are being searched in the line he suggested be removed. Therefore, I need to understand the line itself in order to understand his suggestion codewise.

Just cry Uncle! when you get tired of my endless stream of questions. :D

| is the pipe character, it "pipes" the output (stdout) of one command to the input (stdin) of another command

nice is the command to run processes at a lower or higher priority (softer or louder yelling at the kernel for processing attention) 0 is nominal, <0 = less prio, >0 = greater prio

su is "substitute user", not just superuser. in this case, su to user "nobody" and run as the permissions/file access afforded that user

2>&1 means redirect the error output (stderr = 2) to stdout (stdout = 1) - don't get hung up here. just read this as "capture regular and error output" and read about redirection in your shell's man pages.

sed is the "stream editor" - it reads streams of data (mostly text) and modifies it, in this case, if the stream contains the string "Permission denied", delete it

the \ char at the end of the line mean "continue on the next line" - try it in the shell...

% ls \
-> - l

Exactly what part of the code is saying this?

The light clicks on. You added the \ when you edited the code. Such a simple thing. :p

You are right on the command running twice one night Vicky. You can either:

1) comment out the weekly one

2) have your daily script check if it is Saturday morning (Mine seems to run at about 04:32), and not run the daily one then.

Option 1) is easier, but you will learn more scripting via 2)
8-)

I'm not the one who thought of it running twice. Merv did. I'm the one with the confused look on her face, reading a book several chapters ahead, eating pizza, and asking all the durn questions. :p

Still, I vote for option 2. Never gonna learn if I don't find things to muck with. :D

Re: Re: Updating locate.updatedb daily instead of weekly
 

Okay, we don't have to be nice. Understood. So take out nice -5.

And we don't to index only public files so we must remove su -fm nobody, right? That's the part that's limiting the search to public files.

But Doug has also removed 2>&1 which is what is creating the database in the first place, isn't it? So how can that part be removed and still have the script build the database? I don't think it can. I don't think that part can be removed from the code. No offense, but I think Doug has an error in the code he suggested in his earlier post.

Just a quick mind bender (from the nice manpages):
If you don't want to be nice, leave the nice -5 in there. Basically you're removing nice-ness by setting it to negative.

merv, I think you had it backwards.

If you make something nice -5, it is less nice, and therefore it will be more piggish at the CPU-buffet bar, cut other processes off in traffic, and making rude gestures at everyone.

But then, I could have it backwards. This may be the wrong sort of nice. You never can tell with nice.

doug, still, after the weekly script runs, your locate database is back to what user nobody has access to (other=rx)

what i opt for is a way to trick the weekly script. this is a bit contrived, but trix are for kids...

one thing that these [system maintenance scripts] are missing is a pre-amble. we have the post-amble in scriptname.local, thank you apple, but if we had a pre-amble, we could do things to affect the behavior of the vanilla script run.

this technique requires a mod to the vanilla script, but i think it's modular and relatively easier to retro-fit into "update" scripts than modifying the vanilla code.

if we modify /etc/weekly and insert a self-contained call out to
/etc/weekly-preamble, then we can do some things to trick the code in weekly.

a downside to this is that the locate database is unavailable for the duration of the weekly maint run. meh.

but this assures that the daily.local script's work is maintained and the processes are efficient rather than redundant.

next, we'll explore how to set a trap to insure that the postamble weekly.local gets run to mv the database back, in case /etc/weekly gaks.

also, might be good to log some messages in the console or system.log or both about what's happening. good to leave a trail of crumbs with this kind of stuff.

yeah, i got the nice increments backwards. and this is a sh script, so no csh involved. why would weekly nice this to run louder?

oh, wait. an even better way i didn't see before...

instead of renaming /usr/libexec/locate.updatedb to
/usr/libexec/locate.updatedb.hide , just remove execute access to trip the weekly script's if [ -x ... ]

chmod a-x /usr/libexec/locate.updatedb

and

chmod a+x /usr/libexec/locate.updatedb

instead of the renaming

no downside?

Even I followed your last suggestion (tripping up the if -x), and I like it. Simple and without modifying the standard script. :)

okay, so the new approach to this without modifying the vanilla weekly is:

% sudo chmod a-x /usr/libexec/locate.updatedb

that will make /etc/weekly skip the /usr/libexec/locate.updatedb run

but, our daily.local won't be able to execute /usr/libexec/locate.updatedb by name, right? because, now, it's not executable.

so, we might have to fiddle and test the code for our daily.local

echo /usr/libexec/locate.updatedb | nice -5 su -fm nobody 2>&1 | \
sed -e '/Permission denied/d'

and test it considerably. since /usr/libexec/locate.updatedb is a csh script, perhaps tell csh to run it?...

csh /usr/libexec/locate.updatedb 2>&1 | sed -e '/Permission denied/d'

we'll get to what 2>&1 means later. for now, think of it as "capture all output". it is not what creates the database. that's done entirely in the code of
/usr/libexec/locate.updatedb

Can't /usr/libexec/locate.updatedb remain -x all the time, only to be +x at the beginning of the daily.local, and then turned back to -x at the end of daily.local? That way it is always -x except when daily.local is running. That would trip the if statement in the weekly script to bypass that part without any modification.

Is there anything else using /usr/libexec/locate.updatedb other than just weekly and our new daily.local?

vicki, there you go. well done **

nothing else that i know of runs the updatedb, and if it does, then it'll obvious what the problem is. but the solution will be alluding unless we know that we've done this, right? best to leave a lot of notes and documentation about what we're doing here, like the url to this thread :D

but, this also works well on the un-exec'd script

% sudo csh /usr/libexec/locate.updatedb

Couldn't you check what day it is in the daily.local script? something like
basically, check if it's Sunday(I think the weekly runs Sunday) if it ain't run the update_locatedb for the daily.local. This way you aren't modifying files perms and such. Just a thought. Oh, and please fix my code if it's wrong, I didn't really test it or anything.

Ok, I was backwards with the nice stuff! 'nice -5' will make it chew more CPU and be not nice (actually the csh nice works the same way if you read the man page, don't know what that BUGS section is on about nkuvu, maybe it is just an artifact from the olden days..). Hmm, maybe I was up late at one time and found the command was grinding things to a halt.... anyway....

As for the '2>&1', probably a (nother? 8-) typo on my part.

The reason that is there is because when the script runs as 'nobody' it tries to read lots of directories that it does not have permission to read, hence there are a lot of messages saying 'Permission denied' that go out to stderr.

As an aside, in UNIX all processes get started with 3 input/output 'files'. There is stdin (0) which is the input to the program. There is stdout (1) which is the normal output of the program. And there is stderr (2) which is the error output of the program. stderr is there as a way to get error messages out even though you output is redirected (or piped) to another program. Try this:Now, the file kkkk does not exist and in:
Case 1) The error message about the file kkkk not existing gets sent to stderr and not piped through to the grep command which is why we see it.
Case 2) (go into 'sh' as this does not work in csh's) The stderr and stdout are piped to the grep cammand which doesn't matches fishheads hence no output.
Case 3) again stderr is piped with stdout but the grep catches the error text this time.

So, there will be lots of 'Permmision denied' messages coming out of the update script when it is run by 'nobody'. Now you could just redirect stderr to /dev/null, but then you might miss a 'real' error of some other type. So the '2>&1' says to send the stderr (2) out to the same place as stdout(1) and then we will pipe the whole lot through sed and delete all the permission denied messages. Any other errors will still show up and end up in the email that these scripts send to root (which apple's default setup ends up sending to /dev/null anyway, but that is another story!)

Whew. Now, since I am no longer running this as 'nobody', I am running as 'root' so I will never see those 'Permission denied' errors because I can see everything on my box. So in effect it does not make any difference to me.

As for Merv's catch on the fact that the weekly will produce different output than the daily and the fixes for that etc. that is right on. I guess that is why I just went ahead and hacked the weekly script. 8-)

I think I have just thought of a good project for myself. Some type of database that lets you list a number of files (and maybe stores the originals and check sums of the new versions) and it lets you type in some text describing why you hacked the files. Then after any systems upgrades you could run a check for anything that it 'reset' back to normal and investigate if you still need your hack. Come to think of it I suppose I could use CVS for this.... hmmmmm.

Doug, thank you for explaining it to me! That helps quite a bit. :)

I've decided it's best to put a warning on my sig line so I don't confuse others since I'm more wrong than right half the time!