How many commodity indices does the world need? At present there are at least seven on offer, each used to generate further suites of sub-indices. Among the providers there are some very big names jostling for market share: Dow Jones-AIG, S&P, which bought the Goldman Sachs Commodity Index in January, not to mention newer arrivals Reuters Jeffries CRB, UBS Bloomberg and Deutsche Bank. There are several more distinctly US brands, hardly used in Europe, notably Commins and Rogers International. Providers launch commodity indices for brand recognition purposes, or charge for them as part of broader families of equity, bond and other indices.

Crucially, these indices are a rich source of income from the licences sold to the providers of unitised and exchange traded funds (ETFs), which replicate their composition and price movements. The potential rewards from owning a widely used index are huge.

Nobody can claim to give really authoritative data on the amounts of money in commodity futures and swaps, but best estimates are that funds passively replicating the movement of one or more of these indices currently attract around $100bn (€74.9bn). Some 95% of this total is presently invested in funds replicating either the DJ-AIG or the S&P GSCI of which about one third is benchmarked to the DJ-AIG and two thirds to the S&P GSCI. The true size of the commodities market if measured by outstanding futures and swaps is far larger but not known for sure. The problem with measuring this universe is systemic; most of the relevant futures contracts and swaps are traded over the counter (OTC) and go unmeasured except by each side of the transaction.

All the parties participating in the commodities markets are there for profit, but profit of different kinds.

Corporates seek to hedge their balance sheet risk, reduce their cost of capital and thereby improve bottom line profit. Some investors want passive exposure to one or more commodities as part of a strategy to diversify their portfolios. Others, notably hedge funds and CTAs, trade commodities for pure alpha. As with currencies, it is possible to argue that there are constant inefficiencies in a market this large and liquid with such a diverse range of investors. For pension funds the most eye catching characteristic of commodities is the negative to very low correlation of the leading commodity indices to other major asset classes such as listed equities or bonds.

With so much money behind the two leading indices their construction deserves more detailed examination. The S&P GSCI was the first to be launched in 1991. It seeks to approximate the economic value of the amounts of each major commodity in production. The index weight given to each commodity is therefore determined by the economic value put on them in the real world. "Goldman Sachs no longer runs the index, but I would still defend it's construction as more intellectually justifiable than that of many others," says Michael Johnson, portfolio manager on the fixed income team at Goldman Sachs AM in New York.

Production weighting is defined as the average world production quantities over the past five years. The index has five constituent sectors; energy (75% of total as January 2007), industrial metals with 9%, precious metals 2.1%, agriculture at 9.9% and livestock on 4%. Each these divides into sub-sectors, of which there are 24 in total. Energy, which includes crude oil (31.6% of total), Brent crude(15%), unleaded gas (8.9%), heating oil (8.9%), gas oil (4.4%) and natural gas (6.9%).

By contrast, DJ-AIGCI imposes a priori limits on each commodity sector regardless of whether their production values exceed these limits; none can exceed 33%. It has the same set of five main commodity sectors but these sub-divide into 19 sub-sectors. The most obvious difference with the S&P GSCI is that as energy comprises only 33% of the index by value this increases the allocations to the other commodity sectors. Industrial metals account for 18.2% of the index, precious metals for 8.2%, agriculture no less than 30.2% and livestock for 10.5%.

The arguments for and against each index are fundamental in character. Proponents of the S&P GSCI argue that its construction and energy weightings optimise the correlation benefits of investing in commodities. The rationale behind this is that in an overheated economy, where financial assets begin to underperform, commodities like oil will outperform.

Commodities that make up a smaller share of the index are less likely to outperform relative to energy. The argument in favour of the DJ-AIG, or any other capped weight index can only be based on the relative diversification resulting from these caps. "Our index gives more diversified exposure," argues John Prestbo, managing director at DJ-AIG. "The asset allocation decision is a form of active risk built into the index."

Needless to say, there is a lot of proprietary data on this subject. S&P GSCI has measured their index correlation and that of the DJ-AIG relative to the S&P 500. Over 10 years the S&P GSCI shows a negative correlation of -0.01, the DJ-AIG a positive correlation of 0.10. S&P GSCI also argues that the energy sector of its index shows a negative correlation of -0.04 to the S&P 500 against a positive correlation of 0.20 between the non-energy sectors of the index with the S&P 500.

There are other more technical but important differences between each index. S&P GSCI rolls its energy futures every month, precious metals every other month and base metals every month. By contrast, DJ-AIG rolls energy every other month, silver and gold on alternate months and base metals five to six times a year.

These differences impact on the transaction costs and relative volatility of each index. The most important difference lies in the frequency of the energy rolls. Energy has been in contango for some years. As a result, both indices and the S&P GSCI particularly have been effected by the negative roll yield. Contango describes a state where the price of a commodity for future delivery is higher than the current price. The term backwardation is used to describe a contrary state, where current prices are higher than future ones.

The S&P GSCI has incurred high costs because contango implies that when rolling contracts, the new contracts must cost more. "Nevertheless we tend to prefer the S&P GSCI," judges Alistair Lowe, of State Street Global Advisers. "It is more volatile but has a lower correlation with other asset classes."

Some of the new commodity indices are designed to avoid these problems by effectively extending the forward duration of the contracts rolled to avoid contango. They also make capped or fixed allocations to sets of commodities intended to refine the volatility and return characteristics of both the S&P-GSCI and DJ-AIG. Their designs are back-tested and accompanied by data seeking to prove that they offer more efficient exposure to the asset class. The new indices are best understood as commodity portfolios designed to fit in larger portfolios of other assets, notably equities and bonds, rather than attempts to capture the underlying economic value of the commodity universe. The focus of all these designs is portfolio diversification.

For example, the Deutsche Bank Liquid Commodity Index-Optimum Yield (DBLCI-OY) tries to accomplish a wide range of objectives. It includes only six commodities, crude oil (35% of the index) heating oil (20%), aluminium (12.5%) gold (10%), corn (11.25%) and wheat (11.25%). Each of these is selected to optimise their low to negative correlation with other asset classes. This is a radically different portfolio to the S&P GSCI with far higher weightings to metals and agriculture. "The design also tries to negate the consequences of cotango in the oil market by extending the average duration of energy futures in the index," says Petra Lottig, managing director with global responsibility for commodity products at Deutsche Bank. This is done by permitting the purchase of future contracts going out up to 13 months to optimise yield. The crucial point is that DBLCI-OY no longer uses a pre-set schedule for rolling contracts but permits discretion to roll for the maximum roll yield up to
13 months.

But exposure to the DBLCI-OY can be achieved by buying an ETF licensed by Deutsche Bank, as with all the other indices. In addition to ETFs there are some very large passive unitised funds from the likes of PIMCO, which uses the DJ-AIG, and accepts relatively small investment premiums from institutions. Large pension funds, insurance companies and corporates can also run their own in-house programmes of rolling future contracts or buying swaps via one of the investment banks.

Until recently there was widespread scepticism that commodities could be traded consistently for alpha. Now fund managers are launching funds that have performance targets based either on a relative or absolute return basis. The DWS Active Commodity Alpha fund is a relative return fund using the DJ-AIG CI Excess Return as a benchmark, with a 3.5% outperformance target. "Commodities offer a new source of pure alpha," believes Patrick Armstrong, fund manager at Insight Investment in London, "and there is a strong appetite for this among institutional investors." We can expect more of these fund launches as the battle of the commodity indices grows ever sharper.