The first piece of advice that we got from our business journalism professor in J-school was to forget all economic theories about the markets. It was an odd piece of wisdom. I disregarded it and borrowed the Principles of Economics by Gregory Mankiw from a colleague. I never got around to reading the book. But, I learned the inherent truth of my professor’s advice.

The markets are rarely rational or efficient. Take a look at market performance since the beginning of this year. Even as indicators pointed to a decline in economic health, the markets swung to new highs. “Traders have already priced the decline,” I was told by seasoned experts. However, the markets barely budged, when ECB chief Mario Draghi exceeded its expectations and unleashed a bazooka of stimulus and negative interest rates.

With a recommendation from Warren Buffett, arguably the sharpest investor in the markets for the latest 50 years. Benjamin Graham’s The Intelligent Investor is the definitive guide to stock market investing. But, Graham wrote the book in 1949. This was a time when the market was small and had a bad rep. The book’s premise of a rational market and promise of sustained returns attracted investors and investing grew into a discipline.

Burton Mankiel’s “A Random Walk Down Wall Street” takes the opposite tack. The book’s title is indicative of its intent. It argues that it is not only difficult but impossible to beat the markets. The book’s first edition came out in 1973 and it was revised in 2002 to include a chapter about the dotcom crash. Written in an informative and engaging style, the book is an excellent introduction to the topsy turvy world of investing. It is accessible to the lay reader as well as the informed journalist (Disclaimer: I belong somewhere in between).

Mankiel, who is a professor, starts by providing a brief rundown of major theories in investing. There is the fundamental analysis theory, which relies on a study of the company’s business foundations, such as the market size and its future prospects, to determine long term investment potential. He also outlines four rules for investing and proceeds to demolish them in succeeding paragraphs.

For example, Mankiel shows that projection of future earnings for a company can change depending on the duration of projection of those earnings. As an example, IBM was heavily underpriced, when its earnings were projected forward by ten years back in the 1980s. However, it became overpriced when the duration of projection was extended to twenty years. In reality, the IBM stock drastically underperformed the market during the 1990s and the Armonk-based company was forced to change its business model from product to services to survive.

The technical analysis theory identifies investment opportunities through stock price movements. In other words, it is all about a stock’s momentum. According to this theory, stock prices move in historical patterns and traders can ride the crests and troughs of this pattern to make money. For example, each time the price of a stock reaches “resistance area” (or, the price at which it was bought), a downtrend will occur according to chartists because investors want to break even or book profits. The consequent selling action will drive down its price until traders start buying again.

Mankiel has most fun with this theory. He disparages the theory completely and relates an experiment that he conducted with students. In the experiment, he asked students to construct the chart of a hypothetical stock using a fair toss coin. For each successive trading day, the closing stock price would be determined by the flip of a fair coin. As it turned out, the chart looked fairly similar to a trader’s chart and, according to Mankiel, even displayed similar cycles.

In the following chapters, Mankiel examines other theories and actors related to the business of stock investing. For example, he cites correlation theories that have proved that the indicator most closely correlated with the S&P 500 index is the volume of butter production in Bangladesh. He also puts analysts through the grinder. “To be sure, when an analyst says “buy” he may mean “hold” and when he says “hold,” he probably means this as a euphemism for “dump this piece of crap as soon as possible,” he writes.

According to him, they fail due to an assortment of reasons, including influence of random events, basic incompetence, and conflict of interest. The last-mentioned reason is especially pertinent today, given the plethora of analysts and research firms in the market. Despite the plenty, however, there are very few sell recommendations on Wall Street because companies revoke access to inside information, if an analyst is bearish on its prospects.

All of this then begs the following question:

If the entire apparatus and hijinks of Wall Street is not sufficient to beat the markets, what should the average main street investor do?

As it turns out, Mankiel is a fan of the efficient markets hypothesis. According to this theory, markets are the best and most efficient allocators of capital. In other words, the random movements of a stock security even out over a period of time and the most successful companies generally are the ones that perform the best. He dispenses the same advice as Buffett, who told his wife and trustees to put all their money in index funds after his death. Buy and hold, he counsels investors, instead of buying and selling. This is fair advice for mom-and-pop investors not interested in the daily swings of the stock market.

Except, the markets are not that efficient.

There are a number of examples of individuals and investment firms beating the market. The most famous one, of course, is Warren Buffett. Even as he counseled use of index funds, Buffett acknowledged the possibility of of a couple of investors beating the market. For the vast majority, who do not have the time, expertise and access to engage with the markets, he advised index funds. A number of institutional investors, such as pension funds and hedge funds, have also booked profits by moving in and out of trades quickly. In fact, their popularity and numbers have grown in recent times. According to the research firm HFR Inc, the number of hedge funds in the country has grown from 600 to 10,000.

Then, there is the case for analyzing the fundamentals of an industry. IBM’s dominant position in the computer industry was undercut by Microsoft and Apple and the rise of a number of personal computer companies, such as Dell. Till date, technology is a growth sector and poised to outperform established sectors. For traders, this means that they can ride the coattails of this trend to make money. More recently, a close study of the fundamentals of the 3D printing market would have yielded handsome profits for investors who shorted stocks of major 3D printing companies.

It would seem then that there is money to be made by beating the markets. Which brings us to the question about what really moves markets. Inside information about a stock is one. An ability to parse through jargon in a company’s filings and provide a big picture view is another. There are also certain indicators, as Mankiel acknowledges in his book, that work. For example, trailing P/E ratios are, according to him, good indicators of future performance for a stock.

All of this basically means that the walk down wall street may be random at times but it is by no means completely impossible and uncertain.