Information Processing

Just another WordPress.com weblog

Archive for the ‘iq’ Category

Wonderlic fun

with 12 comments

The Wonderlic is a simple intelligence test widely administered to job candidates, most famously by the NFL in its annual draft. Try this sample test and report your score in the comments 🙂

Note: actual test is 50 problems over 12 minutes, so ESPN is probably incorrect to give 5 minutes for 15 questions. Give yourself 3.6 minutes and multiply your score by 50/15 = 3.33 to get a Wonderlic score, W. Wikipedia estimates: (very roughly) IQ = 2*W + 60

ESPN: …Each year, about 2.5 million job applicants, in every line of work, take the Wonderlic. The average NFL combiner scores about the same as the average applicant for any other job, a 21. A 20 indicates the test-taker has an IQ of 100, which is average.

Some people disagree with the whole idea of IQ testing because they believe the tests are culturally biased and inaccurate. But Charlie Wonderlic doesn’t make grand claims for the score derived from his test. “What the score does is help match training methods with a player’s ability,” he says. “It could be a playbook — what is the best way to teach a player a play? On the field, the higher the IQ, the greater the ability to understand and handle contingencies and make sound decisions on the fly.”

Yes, we know painfully well how “discredited” IQ tests are. But, evidently, many employers and virtually all universities, not to mention NFL franchises, think that your IQ score has some predictive validity…

The diagram below illustrates how scores vary by position (created by Ben Fry). Note even the “brainy” O-lineman fall short of the average score for college graduates, 28. The overall US average is 21, implying 1 SD is something like 7 Wonderlic points, so the spread between halfback and left tackle is about 1.5 SD.

From Wonderlic.com: Every year, around the time of the NFL draft, there is a sudden surge of interest in the Wonderlic Personnel Test (WPT). The Dallas Cowboys first used the WPT for the selection of football players in the 1960’s. Today, it is one of the standard measures used by the combine and NFL teams when considering draft picks.

Although Wonderlic does not score the WPT for the combine, nor does it receive score reports, there are several sources that publish all of the draft statistics, including Wonderlic scores.

The average WPT score for player positions in the NFL are the same type of scores used by employers when hiring people for specific jobs. In his book Paul Zimmerman’s “The New Thinking man’s Guide to Pro Football,” Paul Zimmerman has published what he believes to be the average scores for NFL players by position.

NFL Position / Wonderlic Score / Job Title

Offensive Tackle / 26 / Marketing Executive
Center / 25 / Claims Examiner
Quarterback / 24 / Computer Operator
Tight End / 22 / Police Officer
Safety / 19 / Butcher
Middle Linebacker / 19 / Hospital Orderly
Cornerback / 18 / Machine Operator
Wide Receiver / 17 / Laboratory Assistant
Fullback / 17 / Dock Hand
Halfback / 16 / Material Handler

Of course, the average score by player position is a group statistic. An employer would use an individual player’s score to determine potential job success. As a point of reference, the average score across the United States is 21, while the average for college graduates is about 28.

Written by infoproc

July 18, 2008 at 8:35 pm

Annals of psychometry: IQs of eminent scientists

with 31 comments

I recently came across a 1950s study of eminent scientists by Harvard psychologist Anne Roe (The Making of a Scientist, published in 1952). Her study is by far the most systematic and sophisticated that I am aware of. She selected 64 eminent scientists — well known, but not quite at the Nobel level — in a more or less random fashion, using, e.g., membership lists of scholarly organizations and expert evaluators in the particular subfields. Roughly speaking, there were three groups: physicists (divided into experimental and theoretical subgroups), biologists (including biochemists and geneticists) and social scientists (psychologists, anthropologists).

Roe devised her own high-end intelligence tests as follows: she obtained difficult problems in verbal, spatial and mathematical reasoning from the Educational Testing Service, which administers the SAT, but also performs bespoke testing research for, e.g., the US military. Using these problems, she created three tests (V, S and M), which were administered to the 64 scientists, and also to a cohort of PhD students at Columbia Teacher’s College. The PhD students also took standard IQ tests and the results were used to norm the high-end VSM tests using an SD = 15. Most IQ tests are not good indicators of true high level ability (e.g., beyond +3 SD or so).

Average ages of subjects: mid-40s for physicists, somewhat older for other scientists

Overall normed scores:

Test (Low / Median / High)

V 121 / 166 / 177

S 123 / 137 / 164

M 128 / 154 / 194

Roe comments: (1) V test was too easy for some takers, so top score no ceiling. (2) S scores tend to decrease with age (correlation .4). Peak (younger) performance would have been higher. (3) M test was found to be too easy for the physicists; only administered to other groups.

It is unlikely that any single individual obtained all of the low scores, so each of the 64 would have been strongly superior in at least one or more areas.

Median scores (raw) by group:

group (V / S / M)

Biologists 56.6 / 9.4 / 16.8
Exp. Physics 46.6 / 11.7 / *
Theo. Physics 64.2 / 13.8 / *
Psychologists 57.7 / 11.3 / 15.6
Anthropologists 61.1 / 8.2 / 9.2

The lowest score in each category among the 12 theoretical physicists would have been roughly V 160 (!) S 130 M >> 150. (Ranges for all groups are given, but I’m too lazy to reproduce them all here.) It is hard to estimate the M scores of the physicists since when Roe tried the test on a few of them they more or less solved every problem modulo some careless mistakes. Note the top raw scores (27 out of 30 problems solved) among the non-physicists (obtained by 2 geneticists and a psychologist), are quite high but short of a full score. The corresponding normed score is 194!

The lowest V scores in the 120-range were only obtained by 2 experimental physicists, all other scientists scored well above this level — note the mean is 166.

My comments:

The data strongly suggests that high IQ provides a significant advantage in science. Some have claimed that IQ is irrelevant beyond some threshold: more precisely, that the advantage conferred by IQ above some threshold (e.g., 120) decreases significantly as other factors like drive or creativity take precedence. But, if that were the case it would be unlikely to have found such high scores in this group. The average IQ of a science PhD is probably in the 130 range, and individuals with IQs in the range described above constitute a tiny fraction of the overall population of scientists. If IQ were irrelevant above 130 we would expect the most eminent group to have a similar average.

Conversely, I think one should be impressed that a simple test which can be administered in a short period of time (e.g., 30 minutes for Roe’s high-end exams) offers significant predictive power. While it is not true that anyone with a high IQ can or will become a great scientist (certainly other factors like drive, luck, creativity play a role), one can nevertheless easily identify the 99 percent (even 99.9 percent) of the population for which success in science is highly improbable. Psychometrics works!

The scores for theoretical physicists confirm an estimate made to me by a famous colleague many years ago, that only 1 in 100,000 people could do high level theoretical physics.

Feynman’s 124: in this context one often hears of Feynman’s modest grade school IQ score of 124. To understand this score we have to remember that typical IQ tests (e.g., administered to public school children) tend to have low ceilings. They are not of the kind that Roe used in her study. One can imagine that the ceiling on Feynman’s exam was roughly 135 (say, 99th percentile). If Feynman received the highest score on the mathematical portion, and a modest score of 115 on the verbal, we can easily understand the resulting average of 124. However, it is well known that Feynman was extremely strong mathematically. He was asked on short notice to take the Putnam exam for MIT as a senior, and received the top score in the country that year! On Roe’s test Feynman’s math score would presumably have been > 190, with a correspondingly higher composite IQ.

I thought I should put this post up now, as the new book by Malcolm Gladwell, Outliers: Why Some People Succeed and Some Don’t is out soon and will surely handicap the discourse on this subject for years to come 🙂

Written by infoproc

July 7, 2008 at 4:02 pm

Higher education and human capital II

with 8 comments

I thought I’d also link to some interesting data from a paper by UT Austin economist Daniel Hamermesh, discussed here on the NYTimes Freakonomics blog.

Their survey covered UT Austin alumni between the ages of 23 and 43, revealing enormous variations in average earnings between different majors. Not surprisingly, the business majors and engineers tend to be high earners. However, the highest earners of all are the Plan II (honors college) alumni, who have by far the highest average SAT scores (1364). Note the huge variance within each major (SD = number in parentheses below the mean value), in particular for business, natural sciences and Plan II. For these majors the standard deviation is larger than the mean, suggesting, perhaps, that a few millionaires (startup founders, entrepreneurs?) are skewing the results.

The effect of college curriculum on earnings: …Clearly, there are large differences across major in average earnings, with the highest-earning majors (Honors Plan II, and “Hard” Business) having averages almost three times that of the lowest (Education). Much of the differences across majors must be due to differences in what the students bring to and do at the University. Students in the higher-earning majors generally have higher SAT totals upon entry, and the fractions of students taking upper-division math and science courses and doing well in them are greater too. The differences are also consistent with the results of differential effort in the labor market and male-female differences in earnings. Thus respondents in the higher-earning majors tend to state that they work longer hours than those in lower-earning majors; and except for the Honors Plan II major, the fraction of women in the higher-earning majors is lower. On the other hand, advanced degrees are more prevalent among those graduates who have majored in subjects that eventually generate lower earnings. Family incomes in the areas where the students attended high school do not differ across majors …

…Even within major, taking more upper-division science or math courses and doing better in them raise eventual earnings. While the effects are not highly significant statistically, the t-statistics generally exceed 1.28. A student who takes 15 credits of upper-division science and math courses and obtains a B average in them will earn about 10 percent more than an otherwise identical student in the same major … who takes no upper-division classes in these areas. There is clearly a return to taking these difficult courses. This holds true even after we have adjusted for differences in mathematical ability by using the total SAT score … . The importance of access to this information should not be underestimated. Estimated earnings differences across majors are substantially higher (e.g., the premium for “hard” business rises to 64 log points, that for engineering to 50 log points) when the information on science and math courses is excluded from the equation in Column (1).

Click on the figure below for a larger, more readable version.


Written by infoproc

June 29, 2008 at 8:15 pm

Higher education and human capital

with 14 comments

What good is higher education? The conventional view is that, in addition to producing a well-informed citizenry, it builds important human capital and raises national productivity. But what is the evidence for these assertions? In policy debates we are typically presented with faulty logic: workers in desirable, high value-added jobs (e.g., at Google or Biogen) tend to have lots of education. Therefore, if we want Americans to have such jobs we had better expand access to higher education. The counter argument, that returns to society as a whole from education diminish as access increases beyond the cognitive elite, is given below by a well-known curmudgeon and psychometric realist:

Brutal, just brutal: …There is no magic point at which a genuine college-level education becomes an option, but anything below an IQ of 110 is problematic. If you want to do well, you should have an IQ of 115 or higher. Put another way, it makes sense for only about 15% of the population, 25% if one stretches it, to get a college education. And yet more than 45% of recent high school graduates enroll in four-year colleges. Adjust that percentage to account for high-school dropouts, and more than 40% of all persons in their late teens are trying to go to a four-year college–enough people to absorb everyone down through an IQ of 104.

Note the claim is not that benefits from higher education are zero for the average student, but merely that they diminish significantly as we expand access. At some point we need to consider whether the marginal cost exceeds the marginal benefit. No amount of schooling will turn an average student into a materials engineer, tax lawyer or derivatives trader.

I’m afraid these kinds of thoughts lurk in the minds of most professors these days — I’ve heard them discussed many times. Why can’t my students write? Why can’t my students do simple math? Does the bottom half of the class really absorb anything from my lectures? Is science just too difficult for some people? If I showed you some of the emails I receive from students in my physics 101 course, you would cry at the lack of mastery of grammar and spelling, let alone physics.

Below I excerpt some depressing results from researchers at Stanford and Yale, which support the sorting and signalling model of higher ed, rather than the human capital building model.

Education and Verbal Ability over Time: Evidence from Three Multi-Time Sources

Nie, Golde and Butler

Abstract: During the 20th century, there was an unprecedented expansion in the level of educational attainment in America. Using three separate measures, this paper investigates whether there was a concurrent increase in verbal ability and skills. Changes in verbal ability in the general population as well as changes in the verbal ability of graduates of different levels of education are investigated. An additional investigation of how changes in the differences between males’ and females’ educational attainment are associated with changes in differences between their respective verbal abilities follows. The main finding is that there is little evidence that the large increase in educational attainment has resulted in an increase in any of the measures of verbal abilities and skills.

From the paper:

The results from using these three different measures of verbal ability and skills all show the same striking patterns: (1) there is no increase in scores in the overall population over time; (2) as the number of people obtaining a certain level of education increased, the verbal ability of those terminating with that degree has decreased. …

Comment for the psychometric cognoscenti: where is the Flynn effect here? I see no overall increase in verbal IQ.

See also this less technical exposition:

Nie and Golde: …Our initial hypothesis was that if amount of schooling causally affects any outcome, it would be verbal ability. The vast expansion of the American education system over the course of the 20th century served as our test bed. We expected that the huge increase in educational attainment in the U.S. across the decades would be accompanied by a substantial improvement in verbal abilities. To our initial amazement, we found no evidence for such improvement.

We started our investigation by showing that there is, indeed, a strong correlation between education and verbal ability. The data on which our analyses are based came from the General Social Survey, a program of in-person interviews that has been conducted regularly since 1972 by the National Opinion Research Center at the University of Chicago. While the samples were nationally representative, to avoid complications caused by changing demographics and questions about the validity of such tests with minority and immigrant populations, we included only the native-born, white American population 30 to 65 years of age, using information collected over the last 35 years of parallel surveys. (We used only those 30 years or older to ensure that we were dealing only with people who had completed their education; we stopped at age 65, lest we contaminate the analysis by differential mortality rates.)

Education levels and scores on a vocabulary test given to subjects are indeed correlated (see Figure 1). Over the three-plus decades studied, those with more education got better vocabulary scores, and vice versa.

Those results, however, do not necessarily imply that education causes increased verbal ability. If education did increase verbal ability, we would expect increasing levels of education over time to bring about measurably higher levels of verbal ability. During the 20th century, there was an unprecedented expansion in the levels of educational attainment in the U.S. The average American born between 1910 and 1914 received a bit more than 10 years of education. The average American born between 1970 and 1974 received 14 years of education. In 60 years, the “average American” went from being a high school dropout to having two years of college — a remarkable increase. The increase in education is across the board. A person born between 1910 and 1914 who obtained some postgraduate education was in the top 6 percent of his or her cohort in terms of education. By the 1970s, nearly 16 percent of the birth cohort had some postgraduate education. The percentage of college graduates or beyond has almost quadrupled over the same period, from just over 10 percent to almost 40 percent.

But, as Figure 2 shows, even though education has increased considerably through the decades, and even though education is correlated with verbal ability, verbal ability has stayed practically constant over time. The lack of change in the average vocabulary score of Americans, despite the large increase in the population’s average years of schooling, is an intriguing finding. …

Written by infoproc

June 28, 2008 at 2:51 pm

Asian-White IQ variance from PISA results

with 31 comments

The vexing question of average differences between groups of humans has been the subject of scrutiny for a very long time. Differences in variance or standard deviation (SD) are less well understood, but have important implications as well. This point was emphasized during the Larry Summers debacle, in which he posited that the variance in male intelligence might be larger than for women, even though the averages are similar (more very dumb and very bright men than women). Summers argued that this effect might explain the preponderance of males in science and engineering, even for a very small difference in SD.

Summers NBER speech:
…If one supposes, as I think is reasonable, that if one is talking about physicists at a top twenty-five research university, one is not talking about people who are two standard deviations above the mean. And perhaps it’s not even talking about somebody who is three standard deviations above the mean. But it’s talking about people who are three and a half, four standard deviations above the mean in the one in 5,000, one in 10,000 class. Even small differences in the standard deviation will translate into very large differences in the available pool substantially out [on the tail].

I did a very crude calculation, which I’m sure was wrong and certainly was unsubtle, twenty different ways. I looked at the Xie and Shauman paper-looked at the book, rather-looked at the evidence on the sex ratios in the top 5% of twelfth graders. If you look at those-they’re all over the map, depends on which test, whether it’s math, or science, and so forth-but 50% women, one woman for every two men, would be a high-end estimate from their estimates. From that, you can back out a difference in the implied standard deviations that works out to be about 20%. And from that, you can work out the difference out several standard deviations. If you do that calculation-and I have no reason to think that it couldn’t be refined in a hundred ways-you get five to one, at the high end.

I’ve occasionally heard a variant of the Summers argument applied to Europeans vs Asians (specifically, NE Asians such as Japanese, Koreans and Chinese): although NE Asians exhibit higher averages than whites in psychometric tests (SAT, IQ, etc.), some suspect a smaller variance, leading to fewer “geniuses” per capita, despite the higher mean. See, e.g., this article in National Review:

…The two populations also differ in the variability of their scores. A representative sample of Americans or Europeans will show more variability than will an East Asian sample. In the familiar bell-shaped distribution curve, the bell is much narrower for the Japanese–which is what you would expect from such a homogeneous population.

This difference is a major matter, and it is worth focusing hard on the data. Just about all Western populations report a standard deviation of 15 IQ points. (The SD, a basic measure of variability, quantifies the extent to which a series of figures deviates from its mean.) But the SD for the Japanese and other East Asian populations appears to be a shade under 13 IQ points. That difference does not sound like a big deal, and, in fact, it does not change things much in the center of the distribution. …

…but it does make a big difference at the high end, and it affects estimates of elite human capital availability in different countries.

I’ve never seen any data to back up the smaller NE Asian SD claim. Looking at SAT data shows a larger variance for the Asian-Pacific Islander category, but that is not surprising since it’s a catch-all category that includes S. Asians, SE Asians, NE Asians and Pacific Islanders. I’ve found very little analysis specific to NE Asians, so I decided to produce some myself. I took the 2006 PISA (OECD Program for International Student Assessment) data, which is painstakingly assembled every 3 years by a huge team of psychologists and educators (400k students from 57 countries tested). The samples are supposed to be statistically representative of the various countries, and the tests are carefully translated into different languages. Most studies of national IQ are quite crude, and subject to numerous methodological uncertainties, although the overall results tend to correlate with PISA results.

Below is what I obtained from the 2006 PISA mathematics exam data (overall rankings by average score here). To get the data, scroll down this page and download the chapter 6 data in .xls spreadsheet format. Level 6 is the highest achievement category listed in the data. For most OECD countries, e.g., France, Germany, UK, only a few percent of students attained this level of performance. In NE Asian countries as many as 11% of students performed at this level. Using these percentages and the country averages, one can extract the SD. (Level 6 = raw score 669, or +1.88SD for OECD, +1.28SD for NE Asians.)

OECD AVG=500 SD=90

NE Asia (HK, Korea, Taiwan) AVG=548 SD=95

The NE Asians performed about .5 SD better on average (consistent with IQ test results), and exhibited similar (somewhat higher) variance. (After doing my calculations I realized that there is actually a table of means and SDs in the spreadsheet, that more or less agree with my results. The standard error for the given SDs is only 1-2 points, so I guess a gap of 5 or 10 points is statistically significant.)

Interestingly, the Finns performed quite well on the exam, posting a very high average, but their SD is smaller. The usual arguments about a (slightly) “narrow bell curve” might apply to the Finns, but apparently not to the NE Asians.

Finland AVG=548 SD=80

Returning to Summers’ calculation, and boldly extrapolating the normal distribution to the far tail (not necessarily reliable, but let’s follow Larry a bit further), the fraction of NE Asians at +4SD (relative to the OECD avg) is about 1 in 4k, whereas the fraction of Europeans at +4SD is 1 in 33k. So the relative representation is about 8 to 1. (This assumed the same SD=90 for both populations. The Finnish numbers might be similar, although it depends crucially on whether you use the smaller SD=80.) Are these results plausible? Have a look at the pictures here of the last dozen or so US Mathematical Olympiad teams (the US Asian population percentage is about 3 percent; the most recent team seems to be about half Asians). The IMO results from 2007 are here. Of the top 15 countries, half are East Asian (including tiny Hong Kong, which outperformed Germany, India and the UK).

Incidentally, again assuming a normal distribution, there are only about 10k people in the US who perform at +4SD (and a similar number in Europe), so this is quite a select population (roughly, the top few hundred high school seniors each year in the US). If you extrapolate the NE Asian numbers to the 1.3 billion population of China you get something like 300k individuals at this level, which is pretty overwhelming.

Although it’s all there in the data set, I didn’t have time to examine the male-female variances in mathematical ability (and don’t want to deal with the abuse that might be heaped on me based on what I might find), but I encourage any interested readers to have a look. The authors of the PISA report wisely only reported that male-female averages are similar 😉

Note: as often happens with this kind of topic, a related discussion has broken out at GNXP.

Written by infoproc

June 18, 2008 at 1:08 am

Returns to elite education

with 6 comments

In an earlier post I discussed a survey of honors college students here at U Oregon, which revealed that very few had a good understanding of elite career choices outside of the traditional ones (law, medicine, engineering, etc.). It’s interesting that, in the past, elite education did not result in greater average earnings once SAT scores are controlled for (see below). But I doubt that will continue to be the case today: almost half the graduating class at Harvard now head into finance, while the top Oregon students don’t know what a hedge fund is.

NYTimes: …Recent research also suggests that lower-income students benefit more from an elite education than other students do. Two economists, Alan B. Krueger and Stacy Berg Dale, studied the earnings of college graduates and found that for most, the selectivity of their alma maters had little effect on their incomes once other factors, like SAT scores, were taken into account. To use a hypothetical example, a graduate of North Carolina State who scored a 1200 on the SAT makes as much, on average, as a Duke graduate with a 1200. But there was an exception: poor students. Even controlling for test scores, they made more money if they went to elite colleges. They evidently gained something like closer contact with professors, exposure to new kinds of jobs or connections that they couldn’t get elsewhere.

“Low-income children,” says Mr. Krueger, a Princeton professor, “gain the most from going to an elite school.”

I predict that, in the future, the returns to elite education for the middle and even upper middle class will resemble those in the past for poor students. Elite education will provide the exposure to new kinds of jobs or connections that they couldn’t get elsewhere. Hint: this means the USA is less and less a true meritocracy.

It’s also interesting how powerful the SAT (which correlates quite strongly with IQ, which can be roughly measured in a 12 minute test) is in predicting life outcomes: knowing that a public university grad scored 99th percentile on the SAT (or brief IQ test) tells you his or her expected income is equal to that of a Harvard grad (at least that was true in the past). I wonder why employers (other than the US military) aren’t allowed to use IQ to screen employees? 😉 I’m not an attorney, but I believe that when DE Shaw or Google ask a prospective employee to supply their SAT score, they may be in violation of the law.

Written by infoproc

April 21, 2008 at 3:50 pm

Books and IQ

with one comment

Here’s some research which correlates books with the IQs of their readers 🙂 Now you can check quantitatively whether you have highbrow or lowbrow taste! The method attempts to estimate the midpoint IQ of people who list a particular book as their favorite, using Facebook and university SAT data. It craps out at the really highbrow end, due to low statistics; see below.

Many of the books appearing at the center of the distribution are typically assigned as required reading (A Farewell to Arms, On the Road, A Tale of Two Cities, etc.), hence are likely to be mentioned by low-scoring students who don’t read very many books. Their ranking here is probably deceptively low.

List of schools ranked by SAT (Caltech #1, of course), with links to 10 most frequent Facebook “favorite books” at that university. Click the image below for a bigger one.

Some notable results:

Harry Potter is the most popular book. The Bible is the second most popular book. At least among college students, Harry Potter is, like the Beatles, indeed bigger than Jesus. Harry Potter still wins even if you add “The Bible” and “The Holy Bible” together.

Although I had no idea at the beginning of this project, I was ever so pleased to discover that Caltech is the smartest school in the country (on average).

The smartest religious book is “The Book of Mormon”. The dumbest religious book is “The Holy Bible”. I’m sure this pleases the Mormons immensely.

The dumbest philosophy book is “The Five People You Meet In Heaven” and the smartest philosophy book is “Atlas Shrugged”.

“Lolita” is the smartest book.

The top/bottom 20 books are remarkably stable. I tried 5 different weighting algorithms and their only variation was in the middle. The dumbest books were always at the bottom, and the smartest books were always on top. This is even further corroborated by the fact that the extremes change remarkably little with increasing m.

Do people with SAT >= 1400 just not read books? Yes, they do read books. Just look at those schools’ facebook profiles! However, there often aren’t enough schools with high SATs to have reliable statistics for these high-ringing books. So it goes.

Methodology:

Get a friend of yours to download, using Facebook, the ten most frequent “favorite books” at every college (manually — as not to violate Facebook’s ToS).

These ten books are indicative of the overall intellectual milieu of that college.

Download the average SAT/ACT score for students attending every college.

Presto! We have a correlation between books and dumbitude (smartitude too)!

Books Colleges Average SAT Scores

Plot the average SAT of each book, discarding books with too few samples to have a reliable average.

Post the results on your website, pondering what the Internet will think of it.

Yes, I’m aware correlation ≠ causation. The results are hilarity incarnate regardless of causality. You can stop sending me email about this distinction. Thanks.

Written by infoproc

February 5, 2008 at 5:41 am