Information Processing

Just another weblog

Archive for April 2007

Robot Genius!

with 3 comments

Sorry for the lack of posts — I was traveling all of last week.

Some recent press coverage of Robot Genius:

Network World
Dark Reading
SC Magazine
Network World 2
PC World
Net Security

InfoWorld April 25, 2007: With a name like Robot Genius you wouldn’t expect the company’s leaders to be modest, but the more you hear the firm’s Chairman Stephen Hsu talk about his startup’s new approach to anti-malware, the more you believe the name might fit.

On April 30, the company founded by Hsu and James Hormuzdiar– a partnership best known for building SSL VPN provider SafeWeb and selling it to Symantec for $26 million in 2003 — will formally introduce a trio of behavior-based security products.

By taking a radically different approach to scanning the Web for malware code and using massive computing power to filter out every URL on the Internet responsible for serving up infections, Robot Genius’ technologies will change the way people view anti-virus tools, Hsu said.

“We’re entering an era where the scalability and bandwidth of machines allows for constant monitoring for malware by looking at the entire Web and studying every piece of downloadable software that’s available,” said Hsu. “Meanwhile, the current generation of virus-scanning applications, even behavior-based tools, has reached its limits in terms of finding the most sophisticated attacks, and lacks the ability to adequately fix problems once they’re found.”

Written by infoproc

April 28, 2007 at 5:11 pm

Posted in robot genius, startups

Stata center, MIT

with 5 comments

One of the things I often complain about is that doing fundamental physics is like trying to push a big rock — it’s very hard to tell whether you’re having any effect or making any progress!

Here I am trying my luck with the Frank Gehry-designed Stata center at MIT, home of their Computer Science and Artificial Intelligence Laboratory (CSAIL).

A funny coincidence — in the elevator I noticed a poster for a talk last Friday on blogging and tagging given by an old friend of mine from Berkeley!

Written by infoproc

April 22, 2007 at 9:45 pm

Posted in photos

Vol and tail risk

with 2 comments

Some nice discussion here by financier turned internet entrepreneur Roger Ehrenberg. His blog is on my Google Reader list. See also here.

Volatility Management in a Complacent World

Volatility has a corrosive effect on returns. Two cash flow streams that generate similar average returns, where one is more volatile than the other, can result in sharply divergent IRRs with the less volatile stream yielding the superior result. Therefore, it is clear that volatility reduction has a real and calculable value, but requires a probabilistic view of the world that is often difficult to quantify and harder to pay for. Further, sometimes the pressure for short-term returns can skew rational long-term thinking, causing an increase in the willingness to accept volatility that, in turn, further depresses the value of accepting such risks. And with this the cycle of complacency begins, is reinforced, and feeds on itself until the inevitable happens: event shock. And people will throw up their hands and and say “Look at this tail risk; this is once-in-a-(decade/century/millennium) event.” And those with a keen appreciation for such things will say – I told you so. “Fat tails” and “three sigma events” are now and have always been a relatively ordinary phenomenon. Tail risk is risk that can and should be priced and, depending upon your objectives and stakeholders, actively mitigated. But this requires discipline and cost, two things commonly lacking in those compensated for short-term objectives. And herein lies the issue.

The Economist has an interesting piece titled Sting in the Tail, positing whether low volatility is making the world too complacent about risk. There are some derivatives details in the article that I will address in a separate post that are either unclear or incorrect and, in my opinion, confuse the issue and muddy the picture unnecessarily. However, the questions raised by the article are spot on and important for investors and policy-makers to consider…

1. Have developments in the global financial markets – the rise of derivatives and risk dispersion, stronger, more competent central banks, and a more diversified base of economic power spread across countries, companies and currencies – conspired to reduce the “structural” level of market volatility?

I think the answer is yes. In general, diversification reduces the variability of outcomes, and with the increased diversification of both risks and economic power it must logically follow that some degree of variability has to have been stripped from the global financial markets. This doesn’t mean that markets still can’t get pushed out of whack or that exogenous, non-economic shocks (i.e., war, terrorism, etc.) don’t have profound economic impacts that give rise to volatility. It only means that if one were to take a longitudinal view of global economic performance, that one would expect a less variable set of returns than previously generated in a world with greater concentration of risks and economic power. My only qualifier to this is: in a more complex, diversified world, does this possibly give rise to a third dimension of exposures related to the non-linear increase in the complexity of relationships managing the world’s global economic resources? It was easy to navigate when you had a few economic superpowers and a few powerful financial markets. But what about now? The picture is painfully more complicated. And this might, in and of itself, effect the volatility landscape.

2. Are the perceptions of risk and, therefore, volatility, cyclical?

The answer is unquestionably yes. People and markets have short memories, and that is a fact. Which means that if liquidity is plentiful, spreads are tight and investors are looking everywhere for returns, spreads will continue to tighten as previously risky assets are no longer viewed as risky (i.e., high-yield debt, emerging markets debt and equities, etc.). This also means that those buying insurance are few and those selling insurance are plentiful, further depressing volatility and only amplifying the effects of a complacent market psychology. Then seemingly out of nowhere – bang! Emerging markets debt spreads move from 200 bps to 1000 bps, equity markets drop 10-20%, and risk premia magically expand to meet the heightened anxiety and uncertainty. And this will persist for a while. Until people forget about the shock and it’s once again business as usual. This is how it always has been and this is how it always will be. Will the current liquidity-fueled securities-buying boozer persist indefinitely? Of course not. It is just a matter of time.

3. Are the costs of insuring against tail risk relatively expensive?

Yes. And they always will be. Selling out-of-the-money put options is a risky business, and the implied volatility that is required to buy these instruments is generally very high. In fact, broker/dealers are not the ones best positioned to sell such instruments. While the implied volatility charged the buyer is relatively high the cost of the option is absolutely cheap, due to its out-of-the-moneyness and the low probability of its being exercised. Therefore, the broker/dealer selling the option is not making a whole lot of money even when charging high implied volatility, and the dynamic hedging (usually “delta hedging” – the probability-weighted amount of the hedge underlying that needs to be held based upon the spot price of the underlying and the remaining time to maturity) that needs to take place is subject to “gap risk.” This basically means that when the dealer most needs to sell to adjust its hedge it will be forced to sell at progressively less attractive prices in a market meltdown. This is how dealers can lose hundreds of millions of dollars very, very quickly. However, insurance companies and others with durable, ultra-long dated asset portfolios can sell these options as a vehicle for enhancing returns (the flip side of covered call writing) without the need for dynamic hedging. Bottom line, implied volatility will always be high for these types of instruments, and necessarily so.

4. Can one generate seemingly superior short-term returns by avoiding the costs of insuring long-term risk?

Absolutely. Whether the investor is buying risky assets at progressively tighter spreads or selling optionality as a vehicle for collecting premium it hopes never to have to pay back (and then some), these activities generate returns that are relatively attractive when compared to common benchmarks. However, Mr. Market ensures that investors don’t get something for nothing, so when that “unexpected” shock occurs – spreads blow out, markets drop and margin calls come knocking – the benefits associated with making short-term numbers look good are generally far outweighed by the costs associated with the unwinding of these risky asset/short-option positions. Messrs. Meriwether and Niederhoffer know all about this. So let’s just say that investors should look pretty carefully at fund documents before investing, because often these documents give managers tremendous amounts of latitude, and one will need to dig pretty deep to properly analyze the quality of a fund’s earnings. Is it due to good securities selection or an increase in portfolio risk? This is the question the needs to be answered.

5. Has complacency driven this historical tightening of risk premia across markets, to the point where it is poised to explode in the face of said tail events?

Oh, yes. I think liquidity is a great thing, except when it causes investors to make irrational decisions. Chasing returns. Getting away from a fund’s mission. Assuming risks that are properly absorbed by those better able (and more appropriately positioned) to take them. And when the time comes, no amount of liqudity is going to buttress what looks like an inexorably declining market. Changes in market psychology can be abrupt and harsh. People and governments will move to the sidelines until the detritus is cleared, and this will take exactly how long? Who knows.

So where we are today is at a time when the costs of insurance are both relatively and absolutely low yet the urge is for investors to sell it, not buy it. Because short-term performance considerations (which directly drive most fund managers’ compensation, as well as the ability to gather additional assets to manage) can often drive sub-optimal portfolio decisions. And this is certainly not good for fund investors. And it is at times like these when the smart, savvy, long-term oriented managers with an appreciation for history take a contrarian position. And I might wager that this is precisely what is happening. We’ll see the wheat separated from the chaff in short order. Just wait and see.

Written by infoproc

April 19, 2007 at 3:53 am

Tom Wolfe on hedge funds

with 3 comments

There’s a new glossy business/finance magazine on the racks, called Portfolio. In their first issue they’ve got a profile of Citadel’s Ken Griffin and a long piece by Tom Wolfe on hedge funds. As I’ve said many times here, it’s the new Gilded Age.

I can’t think of anyone better to write this story except possibly Michael Lewis, author of Liar’s Poker.

Twenty years after Bonfire of the Vanities, the author checks in on the new masters of the universe and finds them even coarser and ruder than their predecessors could have ever imagined being.

…First, they have more money, infi-nitely more, than any of the various waves of new money that preceded them, with the possible exception of robber barons on the order of John D. Rockefeller, who, incidentally, was regarded as a rude Pocantico hillbilly Baptist by society in New York a hundred years ago. Hedge funds have what investment managers call “the greatest business plan of all time,” known as “2 and 20.” Each year the typical fund takes 20 percent of the return plus 2 percent of the total investments. Some of the hottest managers, such as Icahn and Stevie Cohen, reportedly take 50 percent of the profits.

In 2005—the figures for 2006 are not yet available—Cohen’s SAC Capital Advisors, of Stamford (he himself lives in Greenwich), made an 18 percent profit on its investors’ $8.5 billion, meaning that Cohen’s income for that one year was in the neighborhood of a billion dollars. This and the figures that follow are the calculations of Trader’s Monthly and Alpha magazine, a niche publication created for the hedge fund industry. No magazine was ever named with greater psychological accuracy, as we are about to see.

Neck and neck with Cohen were James Simons of East Setauket, New York (between $900 million and $1 billion), and Paul Tudor Jones II of Greenwich (between $800 million and $900 million). Further back in the field, nose to nose at $500 million to $600 million each, were Eddie Lampert of Greenwich and Stephen A. Feinberg and Bruce Kovner of New York City. But all six were far, far behind the old man, T. Boone Pickens, who you’ll recall made $1.5 billion in 2005.

Second, hedge fund managers are possessed by a previously unheard-of status fixation. The bellowing door-banger had that status fixation and then some. In Greenwich such characters are not shoehorned into the same buildings as ordinary rich people, which is to say, those with older and far less money. Given Greenwich’s zoning, these people are not likely to express that status fixation neighbor to neighbor. It comes out in other ways.

…The tales are endless: the hedge fund founder desperate to get his son into one of Greenwich’s socially swell private schools who clips a six-figure check to the first page of the application, witlessly forcing the school to reject both his son and his check or lose all credibility—

The lone-wolf entrepreneur who keeps an old-money matron and charity fundraiser waiting outside his office in Greenwich for an hour, remains reared back in his chair with his feet propped up on top of his desk as she comes in, listens to her pitch with his feet on top of his desk, utters a sum total of two words, “Not interested,” with his feet up on top of his desk, and offers no farewell, not even a Godspeed tap-tap of the shoes on his feet up on top of the desk—

The many of these people who spend entire meetings with eyes cast down at their BlackBerrys, thumbing out text messages to God-knows-what-people elsewhere—

The hedge fund manager who, during a 40-minute meeting, takes four telephone calls from his wife on the subject of a dinner party they’re planning, down to the level of who should sit next to whom, whether to serve the champagne in the new flutes or the art deco bowl-and-stem glasses, whether or not endive works as an hors d’oeuvre or is it a little too bitter?—

The hedge fund managers who hold meetings with their shirttails hanging outside their jeans, like college boys—

The former manager of Tremont Capital Group who came to meetings with the fund’s investors barefoot—

The twinkie wives of these people who arrive at real estate offices seeking to-die-for houses and apartments wearing jeans and stiletto-heel boots, with gotta-be-blond hair streaming down to their shoulder blades, holding a baby on a cocked hip with one hand and a cell phone to the ear with the other while a limousine waits outside, motor running—

The twinkies who have their eggs fertilized by their husbands’ sperm in a laboratory, creating embryos for implantation in the wombs of surrogate mothers who are paid to manufacture children for delivery in nine months, since why on earth should any wife whose husband is worth a billion or even $500 million have to endure the distended belly, bilious mornings, back cramps, not to mention a cramped social life, to end up with her perfect personal-trainer-sculpted boy-with-breasts body she has spent thousands of sweaty hours attaining, ruined … tempting her husband to survey all the little man-eaters out there, including those former wives who used to meet regularly at the Boxing Cat Grill until it burned down, whereas the current wives leave their husbands catatonic before the plasma TV and meet three or four times a week at one local bar or another and drive home in their Hummers and bobtail Mercedes S.U.V.’s, bombed out of their minds, while waiting for the baby to come from the factory—

Whenever such rich gossip is re-peated, somebody invariably says, “Who are these people?”

Written by infoproc

April 17, 2007 at 10:11 pm

Nonlinearity and noisy outcomes

with 2 comments

The Times magazine has a great little summary of some recent social science research, which studied the effects of social influence on judgements of quality. The researchers placed a number of songs online and asked volunteers to rate them. One group rated them without seeing others’ opinions. In a number of other “worlds” the raters were allowed to see the opinions of others in their world. Unsurprisingly, the interactive worlds exhibited large fluctuations, in which songs judged as mediocre by isolated listeners rose on the basis of small initial fluctuations in their ratings (e.g., in a particular world, the first 10 raters may have all liked an otherwise mediocre song, and subsequent listeners were influenced by this, leading to a positive feedback loop).

It isn’t hard to think of a number of other contexts where this effect plays out. Think of the careers of two otherwise identical competitors (e.g., in science, business, academia). The one who enjoys an intial positive fluctuation may be carried along far beyond their competitor, for no reason of superior merit. The effect also appears in competing technologies or brands or fashion trends.

If outcomes are so noisy, then successful prediction is more a matter of luck than skill. The successful predictor is not necessarily a better judge of intrinsic quality, since quality is swamped by random fluctuations that are amplified nonlinearly. This picture undermines the rationale for the high compensation awarded to certain CEOs, studio and recording executives, even portfolio managers. In recent years I’ve often heard the argument that these people deserve their compensation because they generate tremendous value for society by making correct decisions about resource allocation (especially if they sit at the cash nexus of finance). However, the argument depends heavily on the assumption that the people in question are really adding value, rather than just throwing darts. If the system is sufficiently noisy it may be almost impossible to tell one way or the other. We may be rewarding the lucky, rather than the good, and a society with fewer incentives for these people may be equally or nearly equally efficient.

See related discussion of studio executives here, and another related discussion here.

As anyone who follows the business of culture is aware, the profits of cultural industries depend disproportionately on the occasional outsize success — a blockbuster movie, a best-selling book or a superstar artist — to offset the many investments that fail dismally. What may be less clear to casual observers is why professional editors, studio executives and talent managers, many of whom have a lifetime of experience in their businesses, are so bad at predicting which of their many potential projects will make it big. How could it be that industry executives rejected, passed over or even disparaged smash hits like “Star Wars,” “Harry Potter” and the Beatles, even as many of their most confident bets turned out to be flops? It may be true, in other words, that “nobody knows anything,” as the screenwriter William Goldman once said about Hollywood. But why? Of course, the experts may simply not be as smart as they would like us to believe. Recent research, however, suggests that reliable hit prediction is impossible no matter how much you know — a result that has implications not only for our understanding of best-seller lists but for business and politics as well.

…But people almost never make decisions independently — in part because the world abounds with so many choices that we have little hope of ever finding what we want on our own; in part because we are never really sure what we want anyway; and in part because what we often want is not so much to experience the “best” of everything as it is to experience the same things as other people and thereby also experience the benefits of sharing.

There’s nothing wrong with these tendencies. Ultimately, we’re all social beings, and without one another to rely on, life would be not only intolerable but meaningless. Yet our mutual dependence has unexpected consequences, one of which is that if people do not make decisions independently — if even in part they like things because other people like them — then predicting hits is not only difficult but actually impossible, no matter how much you know about individual tastes.

The reason is that when people tend to like what other people like, differences in popularity are subject to what is called “cumulative advantage,” or the “rich get richer” effect. This means that if one object happens to be slightly more popular than another at just the right point, it will tend to become more popular still. As a result, even tiny, random fluctuations can blow up, generating potentially enormous long-run differences among even indistinguishable competitors — a phenomenon that is similar in some ways to the famous “butterfly effect” from chaos theory. Thus, if history were to be somehow rerun many times, seemingly identical universes with the same set of competitors and the same overall market tastes would quickly generate different winners: Madonna would have been popular in this world, but in some other version of history, she would be a nobody, and someone we have never heard of would be in her place.

…Fortunately, the explosive growth of the Internet has made it possible to study human activity in a controlled manner for thousands or even millions of people at the same time. Recently, my collaborators, Matthew Salganik and Peter Dodds, and I conducted just such a Web-based experiment. In our study, published last year in Science, more than 14,000 participants registered at our Web site, Music Lab (, and were asked to listen to, rate and, if they chose, download songs by bands they had never heard of. Some of the participants saw only the names of the songs and bands, while others also saw how many times the songs had been downloaded by previous participants. This second group — in what we called the “social influence” condition — was further split into eight parallel “worlds” such that participants could see the prior downloads of people only in their own world. We didn’t manipulate any of these rankings — all the artists in all the worlds started out identically, with zero downloads — but because the different worlds were kept separate, they subsequently evolved independently of one another.

This setup let us test the possibility of prediction in two very direct ways. First, if people know what they like regardless of what they think other people like, the most successful songs should draw about the same amount of the total market share in both the independent and social-influence conditions — that is, hits shouldn’t be any bigger just because the people downloading them know what other people downloaded. And second, the very same songs — the “best” ones — should become hits in all social-influence worlds.

What we found, however, was exactly the opposite. In all the social-influence worlds, the most popular songs were much more popular (and the least popular songs were less popular) than in the independent condition. At the same time, however, the particular songs that became hits were different in different worlds, just as cumulative-advantage theory would predict. Introducing social influence into human decision making, in other words, didn’t just make the hits bigger; it also made them more unpredictable.

Written by infoproc

April 15, 2007 at 6:06 pm

Inequality and NYC real estate

with one comment

Felix Salmon argues that Manhattan property values have nowhere to go but up! (Last I checked, desirable apartments were over $1k per square foot, or > $1M for a 1000 square foot pad.)

His argument is simple: thanks to growing inequality (nonlinear returns to big winners in fields like finance, entertainment, technology), and the unique desirability of living in Manhattan (he details these ad nauseum in his post — as a professional blogger, I suppose he has a lot of free time :-), there will always be high demand for the limited number of units in the Big Apple. I think he’s right, at least based on current trends.

In the first quarter of this year, the New York City housing market boomed even as the rest of the country saw some nasty falls in house prices. And I suspect that the same trend might continue for quite a while. Partly, that’s because precious few Manhattan homeowners have subprime mortgages. But on a much larger scale, it’s because New York is one of a handful of global cities which are the winners in the location stakes. The set of things you buy when you buy an apartment here can’t be measured in square feet.

At 11:18am this morning, I got an email which told me that the Committee on Global Thought at Columbia University was having a discussion about the economics of climate change. The discussants? Jeff Sachs, Joe Stiglitz, and Nick Stern. Said discussion was happening at 4pm, and was free and open to the public. Of course, I went. I was even fortunate enough to be able to put to Stern directly my single biggest question/problem on the subject of climate change. He gave a great answer – and then Sachs answered the question too, and then Stiglitz gave his answer, and then Stern came back and added to his answer. (I’ll blog it in a minute.) It was a wonderful moment, and I thank New York City for it.

…The climate change event took place one week to the day after I went out for lunch with Nassim Nicholas Taleb, and had a fascinating and wide-ranging conversation with him.

…And because New York is a global town, demand for property here is global as well. Every time the dollar falls, New York property becomes that much more appealing to millions of Europeans and Asians who have visited and dreamed of living here: it’s not even expensive, by London or Hong Kong standards.

I wouldn’t be at all surprised were someone to tell me that Sachs, Stiglitz and Stern were all having dinner tonight with Bill Clinton, maybe at the house of Mike Bloomberg or George Soros. It’s the kind of thing which happens in New York – and in precious few other places. Davos, maybe, once a year. As such people move to New York, other such people follow them here, in a self-perpetuating virtuous cycle.

Taleb says, in his latest book, that there’s no particular reason why New York rose and Baltimore fell. But now it has happened, it can’t be stopped. Baltimore will never again be a leading global city. And – I feel comfortable in saying – New York will never again (not in the next few decades, anyway) be a crime-addled drug den like it was in the 1980s. The road from there to here was not foreseeable. But the road ahead is clear: New York City is pulling away from the pack, and the bigger a lead it takes, the faster it goes.

Of course, Felix’s perspective might change after he has kids 🙂

Written by infoproc

April 14, 2007 at 3:16 pm

Skills shortage in India

with one comment

Thanks to a correspondent who sent me the link. I get the New Yorker in the mail, but strangely the last issue or two seem not to have arrived!

I think the main advantage India has over China is linguistic: thanks to their colonial history and the similarity between Hindi and English (both Indo-European languages with lots of cognates), it is much easier for Indians to operate comfortably in the lingua franca of business and science. While the elite of Indian science and engineering are fantastic, the article below points out that depth is still somewhat lacking. I suspect this is less true for China — university-trained engineers there are relatively close to world standards.

Previous related posts.

New Yorker: James Surowiecki April 16, 2007

The economic transformation of India is one of the great business storiea of our time. As stifling government regulations have been lifted entrepreneurship has flourished, and the country has becom a high-powered center for information technology and pharmaceuticals. Indian companies like Infosys and Wipro are powerful global players, while Western firms like G.E and I.B.M. now have major research facilities in India employing thousands. India’s seemingly endless flow of young, motivated engineers, scientists, and managers offering developed-world skills at developing-world wages is held to be putting American jobs at risk, and the country is frequently heralded as “the next economic superpower.

But India has run into a surprising hitch on its way to superpower status: its inexhaustible supply of workers is becoming exhausted. Although India has one of the youngest workforces on the planet, the head of Infosys said recently that there was an “acute shortage of skilled manpower,” and a study by Hewitt Associates projects that this year salaries for skilled workers will rise fourteen and a half per cent, a sure sign that demand for skilled labor is outstripping supply.

How is this possible in a country that every year produces two and a half million college graduates and four hundred thousand engineers? Start with the fact that just ten per cent of Indians get any kind of post-secondary education, compared with some fifty per cent who do in the U.S. Moreover, of that ten per cent, the vast majority go to one of India’s seventeen thousand colleges, many of which are closer to community colleges than to four-year institutions. India does have more than three hundred universities, but a recent survey by the London Times Higher Education Supplement put only two of them among the top hundred in the world. Many Indian graduates therefore enter the workforce with a low level of skills. A current study led by Vivek Wadhwa, of Duke University, has found that if you define “engineer” by U.S. standards, India produces just a hundred and seventy thousand engineers a year, not four hundred thousand. Infosys says that, of 1.3 million applicants for jobs last year, it found only two per cent acceptable.

There was a time when many economists believed that post-secondary education didn’t have much impact on economic growth. The really important educational gains, they thought, came from giving rudimentary skills to large numbers of people (which India still needs to do—at least thirty per cent of the population is illiterate). They believed that, in economic terms, society got a very low rate of return on its investment in higher education. But lately that assumption has been overturned, and the social rate of return on investment in university education in India has been calculated at an impressive nine or ten per cent. In other words, every dollar India puts into higher education creates value for the economy as a whole. Yet India spends roughly three and a half per cent of its G.D.P. on education, significantly below the percentage spent by the U.S., even though India’s population is much younger, and spending on education should be proportionately higher. …

Written by infoproc

April 13, 2007 at 11:34 pm

Posted in globalization, india