Browser VS Browser - How they scored

[RacerX]

Quote: Originally Posted by Xd In my search for the best browser... My disclaimer is this, I tested each load with a timer and no bias or opinion of outcome and each browser was free of any cache prior.

So how did you pick the browsers? Leaving a browser out would seem to be a form of bias as to the outcome.

Sad that OmniWeb, the oldest Mac OS X browser (and best IMO), was somehow left out of all this. From my experience OmniWeb 4.5 has been as fast as Safari for most tasks/sites (not surprising as it uses the same rendering engine as Safari) but is twice as fast (on my systems) at loading Java laden pages.

I guess only us old-timers to the platform would know about OmniWeb. I started using it with version 2.4 (and 2.7 beta) and have been using it as my primary browser pretty much ever since (I did leave it for a few months at version 4.2 to use Safari, but came back at version 4.5).

Still, to include IE and Netscape (both of which are no longer in production) but not to test browsers which are still being produced seem odd.

Quote: In this scenario Safari is best suited for these sites and the results clearly support that.

By omitting any browser (including iCab) these results clearly don't support a true study of what is available for users today and taint the results posted so far.



Other than the glaring omissions, great job!

[Xd]

Here is the 3rd and final study done in completely random sequence. I will add the 3 scores of each and share the overall total for each browser as well. I am not sure what to expect for statistical analysis but I will give the basic solutions that are apparent.

ESPN Discovery EOnline NatnlGeogra BBC Amazon TOTAL
Camino0.8b / 7.27 5.32 3.37 2.50 2.07 2.14 22.67
Safari1.2 / 16.88 4.71 7.71 4.28 3.30 2.84 39.72
Firefox0.8 / 21.57 5.96 4.70 2.94 3.28 3.70 42.15
Netscape7.1/ 11.55 7.90 6.34 2.81 3.35 3.90 35.85
IExplorer5.2/ 24.64 11.47 7.14 7.20 6.74 6.24 63.43
Mozilla1.6 / 19.50 7.18 6.01 3.17 2.59 4.04 42.49
Opera7.50 / 21.34 9.62 4.76 5.97 3.39 5.65 50.73

Here are the 3 totals combined and I list them in the order they finished from fastest to slowest.

Camino0.8b / 91.04 / roasted the field
Safari1.2 / 120.71 / a solid runner-up
Netscape7.1/ 122.1 / a respectable third place
Mozilla1.6 / 130.79 / heading up or down?
Firefox0.8 / 149.16 / another trench digger but feature rich
Opera7.50 / 155.42 / expected more from opera
IExplorer5.2/ 185.29 / old habits die hard

What really stands out to me as a result of this time consuming study is this, out of the 126 load tests and the 18 pages each browser loaded, camino had the quickest load time 10 out of 18 times. Completely dominating. Safari was next with 3 out of 18 quickest load times. Explorer nor opera ever had the quickest load time. I have also learned that without features that suit our personal tastes many of these browsers such as explorer would and be obsolete. My personal choice of browser is Safari mainly because it is very user friendly and super clean however, if I ever get used to the camino interface C Yaaaaaaaaaaaaa!!!!

[Xd]

Thx for the feedback RacerX. How I chose the browsers is easy, they are all free which is why omniweb was left out. Why I omitted icab is because it did not completely load the web pages such as ESPN. Otherwise it would have been in the study. On a side note for you, the pages it did load, it scored last place on all but 2 load times and on those they were 2nd to last.

[RacerX]

Quote: Originally Posted by Xd ...they are all free which is why omniweb was left out.

You can use OmniWeb free. You are asked to pay (which I have considering all the years I've been using it) but you are not required to pay to use OmniWeb.

Does this mean you'll be redoing the test to include all free browsers? There was nothing about OmniWeb which would have stopped you from testing it. To my knowledge using it unlicensed isn't any slower than licensed.

Or is this an example of your meaning of no bias or opinion of outcome from your disclaimer?

[roncross@cox.net]

Xd, thanks for the last and final data. GREAT JOB! I will try and have results and graphs by tonight after I return from work. Xd, even though you haven't included icab in these results, will you still be able to post their time results as misc? For the report, will you also list a few general conditions of each browser such as run as standard - no tweats, appeareance and java turned on, etc...? If there were any hints that you used to speed up any browsers, please post so that I can put in the final report. Please put the system that you have used for the test such as power book G4, 1GHz processor with 1Byte of Ram. Again, thanks and great job.

From my preliminary perspective right now. Camino and Safari are the fastest when you use them over a longer period of time and when you go to urls where it takes inherently longer to download pages. The test also points out the fact that people need to design better web pages. The design of the ESPN web page is poor while that of Eonline is good. Of course, this is all preliminary.

RacerX don't be discourage. Anyone is free to run any experiment they want here. I think that these urls and browsers are a first good choice. I would have personally like to see how this measured up to using lynx on unix. But hey, we can always do it again if we like. We are a free and open society.

From a statistical perspective, it is alway a good idea to confirm results. This would mean running a similar test on another machine. If we get the same results twice, then these results would be a solid as a rock.

thx
RLC

[roncross@cox.net]

Quote: Originally Posted by RacerX You can use OmniWeb free. You are asked to pay (which I have considering all the years I've been using it) but you are not required to pay to use OmniWeb. Does this mean you'll be redoing the test to include all free browsers? There was nothing about OmniWeb which would have stopped you from testing it. To my knowledge using it unlicensed isn't any slower than licensed. Or is this an example of your meaning of no bias or opinion of outcome from your disclaimer?

RacerX, no bias of outcome is simply a statistical term that implies the way you run the experiment shouldn't have influence the results in any way to favor or hinder the performance of any of the browsers. The way to ensure this is to ramdomize the experiment. You want to also ensure that there are no dependencies of running the test as it relates to the previous run. In this case XD has done an excellent job. Again, this is a free society and we can always rerun if we choose, preferably on another machine. I just want to get through these results for now and get the report written.

thx
RLC

[RacerX]

Quick follow up...

How did you classify Opera as free ($39 US) and not OmniWeb ($29 US)? Both supply free-modes of using them... with OmniWeb's seeming more free (of ads) than Opera's.

I look forward to your revised (and unbias) results. Hopefully they'll be as complete as the current ones.

Quote: Originally Posted by roncross@cox.net RacerX don't be discourage. Anyone is free to run any experiment they want here. I think that these urls and browsers are a first good choice. I would have personally like to see how this measured up to using lynx on unix. But hey, we can always do it again if we like. We are a free and open society.

Free to run experiments, yes... but free to say that there was no bias or opinion of outcome when browsers were excluded? The given reason for excluding OmniWeb should have applied equally (in a non-bias test) to Opera... but it was included in all the tests.

We are a free and open society, and as such we are free to call into question the conditions of these tests. Has a fair attempt been made? If Xd has just said "I forgot OmniWeb" and I'll rerun the test with it to correct the data", then I would have had a hard time arguing that point (browsers were tested that I had forgotten about, so it would have been understandable). But I was given a reason why OmniWeb was purposefully excluded which should have applied equally to Opera... but wasn't.

You are going to take the time to write up a report with graphics even though errors are present? It was a nice endeavor, but in the end it was flawed. Most importantly, it wasn't (at least originally) fatally flawed. It could have been fixed. Now, I would probably question any fix given the odd nature of the explanation of the omissions.

Quote: Again, this is a free society and we can always rerun if we choose, preferably on another machine. I just want to get through these results for now and get the report written.

Of course on another machine, even if OmniWeb was included, wouldn't help the current results where OmniWeb was omitted. All future results (which included these results) would be tainted by this.

Well, at least terrorism is at a 30 year low.

[Xd]

Hey robcross looking forward to seeing the final graphs. By the way, good point about the system specs. I am using an imac G3 400 mHz 320 mb ram, X 10.3 Panther. Since I completely reinstalled the OS only days ago, no enhancements were made to any browser of any kind. I did clear the cache on each to keep it clean for this study. Just cut and dry. Below is the misc times for icab 2.9.8.

ESPN Discovery EOnline NatnlGeogra BBC Amazon TOTAL
icab 2.9.8 / N/A(espn) 16.47 8.79 6.14 5.33 4.74 N/A(total)

Hey RacerX i just saw your post and wanted to respond to you. I downloaded opera as a freeware. Freeware. It was free with 2 banners as an addware but, free like all the other browsers. When I saw omniweb it was listed as a shareware and thats where I drew the line, if it costs money now or in 30 days I would disclude it. Why? Money for 1 and because it would be like benchmarking basic windows paint against photoshop 8.0. Ridiculous. http://www.apple.com/downloads/macos...es/omniweb.htm
There was no bias or tainted study. If there was a free version it wasn't clear to me and I looked. Apologies since you obviously feel passionately about this. Every version in the study was free. Not that I found every free version. I am not trying to promote anything, this was something I decided to test for myself and post the results for those interested.

[roncross@cox.net]

Quote: Originally Posted by RacerX Quick follow up... You are going to take the time to write up a report with graphics even though errors are present? It was a nice endeavor, but in the end it was flawed. Most importantly, it wasn't (at least originally) fatally flawed. It could have been fixed. Now, I would probably question any fix given the odd nature of the explanation of the omissions. Well, at least terrorism is at a 30 year low.

RacerX please point out the errors that you are referring to with respect the data. I don't want to second guess you. Simply omitting a browser in a test is not an error.

I would consider an error to be one where false conclusions are drawn based on the data that is collected and reported. Let's exclude omniweb for a minute in this discussion. Do you question the validity of the data with respect to the browsers that were choosen (no omniweb don't count)? With the browsers that were choosen, do you see an error in the conclusions being drawn on the data with respect to the browsers in the test? Other then omniweb being excluded from the test, what errors do you see.

Some of the results are as follows:

- internet explorer is still slow. Would running omniweb in the test change this? no, not on your life.

- load times being depended on the url. Would running omniweb in the test change this? no, not likely

- Eonline appears to be a very web designed url. Would running omniweb in the test change this? no, not likely

- Safari and Camino appear to be faster than the rest. Would running omniweb in the test change this? possibily

Ok we missed omniweb, therefore we will have nothing to report on it in our final results. We will have to live without the results of omniweb for now. If we run another test we will be sure to include it. It will NOT be too late, I do statistics and experiments for a living and we are always adding new stuff or leaving something out of a experiment. However, data is never wasted unless someone throws it away. If you want to throw the data away then that is your choice. But based on where I sit, the test is still valid for the other browsers that were choosen.


Please speak to the conclusions drawn in the report. Don't let your emotions get the best of you

thx
RLC


thx
RLC

[yellow]

There are so many variables involved here, it's hard to be scientific. IMO, before the conclusions are posted/"published", these test should be conducted over the course of a month, at a variety of times of day, and then the averages compiled and submitted. That's the only way you're going to get closer to a set of 'real' numbers.

Xd, I am quite glad that you realized we weren't messing with you when you first proposed this and actually took the initiative to bring your idea to fruition. Keep it up!

[Xd]

Actually Yellow I am NOT sure you werent messing w/me, lol. All I know is I don't care. I needed something to chew on. It was a fun and interesting project that allowed me to answer the question for myself as to the overall strength of the INCLUDED browsers. As for the over a month suggestion umm, i dunno. lol

[yellow]

Maybe it's just me then. I would be unsatisfied with the results if I knew that they could be way off the mark. Because of all the variables involved.

[RacerX]

Well, of course I know nothing of mathematics or statistics, so I guess my opinion doesn't hold water here... but lets look at this anyway.

Quote: Originally Posted by roncross@cox.net I would consider an error to be one where false conclusions are drawn based on the data that is collected and reported.

Data was collected or reported incompletely, but a reason for the exclusion of some data (which should be considered a control) was given. However, that same reason should have been applied to other data and wasn't.

Not following controls of this nature would introduce errors in the conclusions which was to find the best browser (later modified to free browser) for our platform on the given system. Are you arguing this point?

Quote: Let's exclude omniweb for a minute in this discussion. Do you question the validity of the data with respect to the browsers that were choosen (no omniweb don't count)?

No more (and no less) than I dispute baseball statistics from before blacks were allowed in the majors. The data is fine if you are comparing any two of those browsers head to head. The data is flawed if you are seeking the best browser.

Are you arguing that point? ...as a professional?

Quote: - Safari and Camino appear to be faster than the rest. Would running omniweb in the test change this? possibily

From this statement I would guess you are not arguing against my assertion.

Quote: Ok we missed omniweb, therefore we will have nothing to report on it in our final results. We will have to live without the results of omniweb for now.

But you do have something to report... that OmniWeb was erroneously omitted when the controls were misapplied. And that the final conclusion is that without OmniWeb there is no final conclusion.

Are you arguing that point? ...as a professional? Do you often omit factors such as that in a report on statistical data?

More to the point... if I had not made a point about making sure that it was reported in the final report, would you have been inclined to make such information known?

Quote: If we run another test we will be sure to include it. It will NOT be too late, I do statistics and experiments for a living and we are always adding new stuff or leaving something out of a experiment.

Only if you do another test where this data is omitted completely. You can not add data from another test (with OmniWeb) to this test (without OmniWeb) and actually be able to say anything conclusive about OmniWeb compared to the other browsers.

Or are you arguing that point? ...as a professional?

Let say that someone else does the test again, on different hardware with all other controls in place and now including OmniWeb. Would you really be able to tell me that you could pool all the data and make any form of conclusion of OmniWeb compared to other browsers?

My, humble, understanding would be that for the given purpose of the data collection, all the data is now tainted as the controls were misapplied. Only rerunning all the tests on all the browsers would solve this issue.

Quote: However, data is never wasted unless someone throws it away. If you want to throw the data away then that is your choice. But based on where I sit, the test is still valid for the other browsers that were choosen.

Yes, like I said, it is good data for head to head comparisons with the chosen browsers. But it is worthless for the original stated purpose of the data collection which was to find the best browser. It should be thrown out with regards to any study of the original purpose.

Are you arguing that point? ...from where you sit?

Quote: Please speak to the conclusions drawn in the report. Don't let your emotions get the best of you

The conclusions of any report on the best browser are erroneous due to the misapplication of control factors. The best we can hope for is a look at head to head comparisons or comparisons of a subset of browsers.

As you are trying to rationalize all this, I would conclude that you're more in danger of letting emotions cloud your judgment here. Why do you wish to push forward when you know there were errors in the testing? Isn't it important to report accurate results? Do you often report results that you know have flaws?

This could easily be done over if accuracy mattered. Had I not pointed this out, the report would have only suffered from human error (someone forgot a browser), but now that this has been brought to light and you wish to push forward anyways, the report is no longer suffering from an error, it is suffering from deception.

Don't let your emotions get the best of you when drafting your report. Remember, your a professional.

[RacerX]

Quote: Originally Posted by Xd When I saw omniweb it was listed as a shareware and thats where I drew the line, if it costs money now or in 30 days I would disclude it.

You never need to pay for it (though if you do like it and do use it that would be nice). OmniWeb can be used unlicensed indefinitely. Instead of 2 banners, you are asked at the startup if you would like to license your copy. If you click "use unlicensed", you go on about your browsing without any other interference from the application.

That sure seems more free than having to deal with addware while browsing. Using Opera for free is losing the ability to block pop-ups by choice.



Maybe that is why Opera cost more than OmniWeb... it is $10 more annoying than OmniWeb's startup dialog box.

[Xd]

RacerX you make a fine debate. My finger is tired from the scrolling. lol. Take it for what it is, my test of these 7 browsers and how they did. I am not claiming camino is the BEST but the best of these 7 for the study done. I am not on the cover of Wired magazine or pulling a jobs and claiming its THE fastest ever. It just performed best in my study. That simple.

oops about your post, honest to god I was not aware and I worked too long and hard to start feeling like bull---- because of this. If I do test again I will include omniweb.

[mclbruce]

Xd, thanks so much for doing this. It's been fun to read the results and other people's comments.

For those that think the testing should be done better or differently: the browsers are still out there, anyone can do their own tests and find their own results. If you feel strongly about this, then go for it!

[Irene]

What a great service you did for the rest of us by compiling this information.

I had been using Mozilla and Safari (with IE for sites that absolutley require it, like my accountant's on line tax workbook) and just downloaded Camino because of your report.

Thanks for sharing.

[RacerX]

Quote: Originally Posted by Irene ...and just downloaded Camino because of your report.

Interesting...



Xd,

My last words on the subject... would this person have been downloading OmniWeb had it taken Camino's spot in the testing?

I'm glad that you are happy with the results, but this test has had an adverse effect on OmniWeb without it getting a fair chance. At least the other browsers had a sporting chance.

I don't want you to feel bad about this, but when you do things like this (so that you are heard and listened to), it does have an effect. I haven't asked anyone to do anything I wasn't willing to do myself. I have often committed myself to learning an application to get it a fair chance and a fair review. I just finish three weeks of learning TIFFany (both 2 and 3) so I could do that.

And no mclbruce, I don't think that a test by me would have any chance of changing the effect of this report as I have already shown my strong support for the Omni Group here.

But as you don't seem to feel strongly about this, I'll be happy (and consider it completely fair) if you were to redo the test on your system... seeing as you brought it up.

[Xd]

Alot of feedback, thats always good. Thx for the kind words. I do apologize for not being able to chart it out cleanly but, Im hoping I will get help with that. RacerX I really appreciate your posts and you gave me more to consider and thats always helpful for future studies. I wish omni was included for the studies sake. This was my desire to learn something new and try and give something back for the endless help others have shared with me. Thx for that. Now if I can learn how to do an html and clean it all up. lol

[mclbruce]

Quote: Originally Posted by RacerX don't want you to feel bad about this, but when you do things like this (so that you are heard and listened to), it does have an effect. I haven't asked anyone to do anything I wasn't willing to do myself. I have often committed myself to learning an application to get it a fair chance and a fair review. I just finish three weeks of learning TIFFany (both 2 and 3) so I could do that. And no mclbruce, I don't think that a test by me would have any chance of changing the effect of this report as I have already shown my strong support for the Omni Group here. But as you don't seem to feel strongly about this, I'll be happy (and consider it completely fair) if you were to redo the test on your system... seeing as you brought it up.

I have no reason to do any testing. I'm satisfied with the performance of my browser and with the tests that have been done. If the results of this test are enough to kill Ominweb, I will happily dance on it's grave.

There is an old song/story about a guy working on a railroad crew. They are out in the middle of nowhere and, while the railroad has supplied a car with food and cooking utensils, they don't supply a cook. The crew decides that the way to select the cook is to make the guy who complains the loudest about the food do the cooking!

http://www.utahphillips.org/stuff/mooseturdpie.mp3