Browser VS Browser - How they scored

[RacerX]

Quote: Originally Posted by mclbruce ... I will happily dance on it's grave.

It is always good to know that social graces haven't been wasted on you.

A little off topic, but can I suggest some reading material for you.

[roncross@cox.net]

First of all, I want to thank Xd for running the test. I know it took him a lot of time to get the data. Both loading the pages and keeping track of time to posting results. It is appreciated.

Please refer to the following website for graphs:
http://homepage.mac.com/ronosx/brows...otoAlbum9.html


According to Xd, the system that he used is as follows:

"This test was done on a G3 400 mHz 320 mb ram,os x using Panther. There were no enhancement made to any browsers of any kind. I clear the cache on each to keep it a clean study."

The 1st two run orders were sequential (see previous post), but the last run order was completely random. I looked at the residuals and there doesn't appear to be residual effect with respect to run order, url, etc... We lucked out this time. In the future it is always a good idea to randomize the data.




Here is the excutive summary.

Based off the graphs labeled "This is the analysis of variance for the browsers during the test" where each measurement is group by test 1, test 2, test 3 as it was collected. The results are as follows:

1.) Camino is clearly number 1 and in a class all by itself. Camino is roughly 30% faster than Safari. Camino took off after the data was completely randomized. Camino has a mean load time of around 5 secs.

2.) Safari, Netscape, Mozilla are group together in 2nd place. This group is roughly 30% slower than Camino. This group has a mean time of around 6.9 sec. I will still use Safari.

3.) Opera and Firefox are the second slowest group and is roughly 72% slower than Camino and roughly 25% slower than the 2nd group. This group has a mean time of around 8.4 sec. I currently use firefox, but will now throw it away for Camino.

4.) Internet Explorer is the slowest of all. It is 100% slower than Camino and 50% slower than the 2nd group. This group has a mean time of around 10.3 sec.


That's the executive summary in a nutshell.

If you want to know about the other graphs, then continue to read, but I do not consider them executive summaries.

Please see the graph labeled "browser speed distribution".

This is the distribution of all the browsers speed put together. Notice that there are some outlyers with an excessive load time - they are darkened in the graph. This comes from the following browser + url combinations. ESPN and firefox, ESPN and IExplorer, ESPN and mozilla, ESPN and opera, ESPN and Safari. This plot demonstrates how a poorly designed website can degrade the performance of a browser. The only browsers that wasn't effected by excessive load times were netscape and Camino. Camino and Netscape is doing something right. What are they doing differently than the other browsers?

Please see the graph labeled "min, mean, max of the browser".

This graph shows how stable each browser is under the test. Camino never takes more than 10 secs to load a page. Although Netscape isn't the fastest, it does appear to be stable, its' maximum value never exceeded 15 sec. In this case, both Camino and Netscape are more stable than the other browsers. The data suggests that Safari has trouble when dealing with poorly designed web pages such as ESPN and that is why its maximum time to load is relatively high. IExplorer sucks, I will not have anything else to say about it here in this post.

Please see the graph labeled "This is the min, mean, max, total load time for each browser"

Camino is again ahead of the pack when total time is taken into account. The maximum time to load all url with Camino is never more than 40 sec. This is an important point since most people generally load different pages from different sites over a long period of time. So this represents a cumulative effect on each person's browsing experiencing. Over a long period of time (6months to a year), you will on average save more time loading web pages with Camino than you will let say using any of the other browsers.

The final data table is labeled "Average speed data table". This is simply the average of the three test runs with respect to browsers and urls. This data has already been mentioned earlier, but it is put in a table for your review. Please take a look at the icab data if you are interested. In this test the icab browser performed poorly and did not load the ESPN page. The lesson here is that a poor browser and poor url don't mix.

There are more observation that were made earlier by me on this forum and I will repeat the results only. If you want explanations of the results, please read the post and look at the data.

Other points already made earlier that must be modified slightly

1.) Camino benefits from the fact that its load time is not significantly degraded when downloading pages from crappy sites. Where other browsers may slow down, Camino keeps on rocking.

2.) Eonline appears to be a very well designed url. There is an important point to make here. The url is as much a function when loading web pages as the browser that is being used. This is based on the two way anova results.

3.) Internet Explorer is a poor browser. Oops, I mentioned it again.


To the forum adminstrator:

My webpage is not a permanent location for these images, so I would appreciate it if someone could put the images in a more permanent location to preserve them for the future.

Thanks for having the patients to go through this and remember to please choose your browser(s) wisely.

[Xd]

Wow, Nice job roncross! I am very pleased with your work. I expected a chart but these are outstanding. I can say without any reservation this was the best experience I have ever had interacting with a forum. This was a good collaboration and I appreciate all of your efforts to make clear what was in dire need of a professional layout. I think it deserves a permanent place in the forum but that is not a decision I can make. I am sure others will appreciate the efforts you have taken as well. Thx to roncross and to all those who shared their opinions as well. And thx to this really cool forum.

[Craig R. Arko]

You could try adding the images as attachments to a post. I have not actually done that myself, but others have and it's worked. I don't know if it's best to put them all in a single post or use a post for each.

You seem to be experimentally inclined, however.

I think the charts and data tables would be more useful if they identified the version of each browser as tested, since some of them change quite rapidly. In the six months we're all saving time most (excepting I.E., of course) will have undergone significant revision.


P.S. - let's keep the 'off-topic' advice to a minimum. We try not to do that stuff here. Thank you.

[roncross@cox.net]

Quote: Originally Posted by Craig R. Arko You could try adding the images as attachments to a post. I have not actually done that myself, but other have and it's worked. I don't know if it's best to put them all in a single post or use a post for each. You seem to be experimentally inclined, however. I think the charts and data tables would be more useful if they identified the version of each browser as tested, since some of them change quite rapidly. In the six months we're all saving time most (excepting I.E., of course) will have undergone significant revision.

Hi XD, hopefully this will be the last request. Will you please provide the version number of each browser?

When I return home, I will try and attach the image in the post itself.

thx
RLC

[Xd]

Hi roncross, really good work! The versions all follow the browsers name. I indluded them for each. Let me know if I can help you out w/anything.

Camino0.8b /
Safari1.2 /
Firefox0.8 /
Netscape7.1/
IExplorer5.2/
Mozilla1.6 /
Opera7.50 /
icab 2.9.8 /

Nice to hear from the site admin also, good tips as always..

[Xd]

I have decided to do 3 more complete tests over a 1 month period of time. Although my original initiative is complete, I do agree with Yellow that this study would be substantially more accurate in terms of true consistency. I will also include omniweb for those who wish the comparison. I would however like some constructive feedback ie opinions that any of you may have in terms of other methods or FREE browsers you would like in the test as well as any other ideas you want to share with this study.

[nkuvu]

Only thing I can think of is adding automation to the whole thing. A relatively long AppleScript, for example, so that you do the same thing every time you run the test. Which you can then distribute so that others can perform the same steps and compare times.

I'm not sure all the browsers in the list are AppleScript-able, though.

[bramley]

Quote: Originally Posted by nkuvu I'm not sure all the browsers in the list are AppleScript-able, though.

Netscape yes
Safari yes
Mozilla yes
Camino er, not sure - I think there's a bug here. No.

and I don't have it but I know Explorer does do Applescript
and the others I don't know.

[roncross@cox.net]

All of these browsers should be scriptable with respect to launching urls. A simple code would look something like the following:

property target_URL : "name of url starting with http"

tell application "browser name"
activate
open target_URL
end tell

This code will open up any browser, this is similar to opening up the browser using the finder.

With respect to a valid statistical test over a longer period, you have to think through very carefully how you want to do this? This should be planned out first before proceeding otherwise you may not be able to draw the correct conclusions. In my opinion, the test could be done in a few days and not months, if we have enough volunteers to help out with test. We would get more data in a shorter amount of times. All you really need to do to ensure a valid test is to make sure that the data is completely randomized and that each run is independent of the other run.

Here is something that I proposed a while back to another group. We might want to try this with some slight modification. I will have to give it some more thought to finalize the details.

Is loading the page based on the amount of RAM that is used and the number of applications that are already open? Does the CPU effects browser speed? We need more data in order to make a true determination. I propose the following test with 5 volunteers.

A quick test would be the following;

Assign A to Safari
Assign B to FireFox
Assign D to netscape
Assign E to OmniWeb
Assign F to Mozilla
Assign G to Lynx
Assign H to Camino

pick two web page such as www.yahoo.com and
http://www.nasa.gov/vision/universe/.../mer_main.html

and run the web pages in the following order below for each person over a two day period. The first day we would run www.yahoo. com and the second day we would run the other web page.

                  mach 1	  mach 2	mach 3	 mach 4	mach 5

browsers FBCGHEAD GABCFHDE EFBAGDCH HCAGBEFD DFABHECG

Since each person mach is slightly different, the data will be independent from the other person system. Since the run order is randomized for each given person, we will be able to eliminate autocorrelation based on applications and RAM already loaded. This will give us some insight into the speed of these browsers. If I can get 5 volunteers, we can run the test, record the results in seconds, and report our findings. It should be fun. I can do the statistical analysis and share with the group.

Any takers,
Ronald Cross


thx
RLC

[Xd]

The same system "must" be used for the complete study and then if another machine were to complete the same study we could compare the results of different systems. Otherwise we slip into the abyss of studying chaos theory. Random machines doing seperate tests would result in random results and therefore have no base to draw a conclusion upon.

[vancenase]

all this fuss over browsers ... and the data is already outdated.

firefox 0.9 was released yesterday

[roncross@cox.net]

Quote: Originally Posted by Xd The same system "must" be used for the complete study and then if another machine were to complete the same study we could compare the results of different systems. Otherwise we slip into the abyss of studying chaos theory. Random machines doing seperate tests would result in random results and therefore have no base to draw a conclusion upon.

This is simply not true. On the contrary, statistical experiments is totally based on randomization and independence treatments. Doing a statistical experiment any other way is simply asking for trouble. A well designed statistical experiment can take in account all of these factors and tell you what is the most vital factors that affects the response -in this case browser speed. This is simply referred to as a screening experiment. The mathematical foundation for this solid and uncomprimising.

Believe me when I say that I have seen a lot of bad decisions made and money thrown down the drain because someone has failed to randomize and ensure independent runs.

Again, I do this for a living so I know what I am talking about. For those that do not practice statistics, some of the concepts may be counter intuitive.
thx
RLC

[Xd]

I do appreciate your professional opinion. In my experiance I have found that you need randomization, it is a necessary condition. However having said that, the only basis for conclusion is in the comparison from the source of a control group. I believe that.

I started this thread as a study to find the fastest browser between the 7 mentioned on my machine and shared the results. I am very satisfied and feel that anyone who duplicated my efforts would find similar if not identicle results. Contrary to my earlier posts, I will pass the baton to others who may help out in this random study. I will enjoy reading the results and will participate if needed. But I will take a back seat to this approach as my ADD is finally kicking in. lol.

[vancenase]

a new mozilla (thunderbird) has been released too!

[edit: i am three-quarters retarded ... read Xd's post below]

[Xd]

I have the new firefox 0.9 ready to go. Looks very slick and is reported to be streamlined for speed. The thunderbird is an e-mail application not a browser. I had intentions of adding the new omniweb 5.0 to the study but it seems that it is no longer freeware (30 trial period). Better hold on to that 4.5 for whoever made the fuss about it earlier, I still have the 4.5 which I will include for future tests if I get the inlination, since it is free.

[RacerX]

Quote: Originally Posted by whoever posted this I had intentions of adding the new omniweb 5.0 to the study but it seems that it is no longer freeware (30 trial period). Better hold on to that 4.5 for whoever made the fuss about it earlier, I still have the 4.5 which I will include for future tests if I get the inlination, since it is free.

OmniWeb 5 is a beta, all the betas of it have expired... even for those of us with licenses. Omni doesn't want people reporting issues with earlier versions of the beta so they made them expire after 30 days.

Currently you can not (even by paying) extend the beta period.

For whoever posted that, if you want to test the Beta, you can... I (personally) tend to use the 4.5 version for daily browsing though.

[Xd]

After reading about the new firefox 0.9, well I had to see for myself. Here are the scores.

ESPN Discovery EOnline NatnlGeogra BBC Amazon TOTAL
Camino0.8b / 9.96 8.57 3.30 3.38 3.02 2.83 / 31.06
Firefox0.9 / 10.19 8.77 5.39 3.67 2.90 3.41 / 34.33
Safari1.2 / 12.8 7.74 5.91 2.37 2.97 3.18 / 34.97
Netscape7.1/ 11.20 9.49 6.50 3.80 3.13 3.57 / 37.69
Mozilla1.6 / 14.74 7.63 6.32 4.27 2.67 3.47 / 39.10
Opera7.50 / 18.54 9.02 5.02 4.97 4.74 3.63 / 45.92
IExplorer5.2/ 17.98 8.68 7.06 6.10 6.31 6.71 / 52.84
Omniweb4.5 / 27.38 14.45 6.18 4.40 4.97 5.44 / 62.82

A few comments on each;

Omniweb4.5 / Performed well except hung on 2 loads otherwise avg speed.
Camino0.8b / Waiting for the beast that can top this one.
Safari1.2 / Really solid browser. Easy to see why its favored by many.
Netscape7.1/ Amazed that this one just hangs in there.
Mozilla1.6 / Good but just not great yet.
Firefox0.9 / This is a fast browser on par w/safari
Opera7.50 / Basically a slumpy browser.
IExplorer5.2/ Hanging by omniweb is all that saves this from the bottom.

[macmath]

Due to this discussion (thanks, roncross, Xd, and others), I downloaded Camino and gave it a try. I like a lot of things about it, including its rendering speed (although it is not enough faster than Safari for me to notice the difference). One thing I do notice on my machine (it might be different with others) is that it takes more horsepower to operate than Safari does. I get spinning cursors that I have to wait for before I can switch tabs; many times typing a post like this has typing delays (like each letter is being squeezed out of a cake decorating tube). Does anyone else notice this? (I finished this previous sentence [Does anyone else notice this] while the display still was beginning to write else.) I think I'll head back to Safari until the next version of Camino or until I can afford a faster machine.

[Xd]

To be honest with you, I have not noticed that yet(fingers crossed). This is why we hoped someone else would do the tests also. I like the look and clean functionality of safari. Explorer was so unresponsive that anything would have been better. I am not a promoter of camino or a basher of other browsers, I really just thought it would be interesting to see which free version works fastest but it should be noted that we are after all talking about mere seconds for the casual user. If you are online all day those seconds turn into 5-10 minutes of "waiting" time and by the end of the day/week it becomes really significant.