Enter your email address below to receive notifications of new posts.

My lastpostdealt with the characteristics and the potential usefulness of the aggregate social media sentiment in the U.S. stock market. As we know, social media is only one of the many possible proxies of theinvestor mood. I will look below at some of the alternative indicators from a quantitative perspective. Well keep the focus in the U.S. where theres a good variety of data available. These are some of the questions well focus on:

Are the different indicators measuring the same thing?

How timely are the various sentiment proxies?

Is there a lead-lag structure between the indicators?

In the interest of keeping this post concise (or lets say not overly long), well compare the social media analytics to one survey of investor sentiment and to implied volatility; well also look at the investor mood vis–viz consumer sentiment. We let other mood variables, newsletter surveys, the trading volume, the dividend premia, IPO activity, insider trading statistics, mutual funds flows, investor positioning, news analytics, and search engine activity go by for now. This is not because any of these indicators or data sets are uninteresting; on the contrary, some are high up on my list.

Before the advent of the social media, researchers used to measure the market sentiment in the old-fashioned way  by asking for it. Robert Shiller1was among the first to systematically survey investors views. His work is being carried on at the Yale International Center for Finance, who maintain various monthlystock market confidence indices. Regular surveys of investment newsletters are conducted by Investors Intelligence and Market Vane. Views of individual investors are canvassed by the American Association of Individual Investors (AAII) and by companies such as Ned Davis Research. The stock market exposure of investment advisors is polled by the National Association of Active Investment Managers (NAAIM).

Ill employ survey data from AAII where the history of the sentiment readings is in the public domain. Bear in mind, though, that the number of responses is often fairly small. It is therefore not evident to what extent the results reflect the sentiment of the wider investment public or that of the institutional asset managers. With that health warning in mind, lets get going. Ill usePsychSignaldata as a measure of the social media sentiment, aggregated at the market level.

AAII conducts weekly polls of retail investors, and theyve been doing it since 1987. The survey runs from Thursday through the following Wednesday, and the percentage of bullish/neutral/bearish investors is published on Thursday mornings. The results are available onQuandlas well asAAIIdirectly. Chart 1 shows the social media sentiment together with the difference between bullish and bearish responses in the survey through time.

Chart 1. S&P 500 index (blue line) vs. the aggregated social media sentiment for the S&P 500 constituent stocks (orange line) and the balance of bullish and bearish investor sentiment in the weekly AAII survey (brown line). Sources: Quandl, PsychSignal, AAII.

There appears to be some similarity between the social media sentiment and the retail survey. This is also evident in a scatter plot (Chart 2), where the social media sentiment is shown against the AAII sentiment.

Chart 2. The aggregated social media sentiment for S&P 500 (x-axis) vs. the AAII survey results (y-axis). Sources: PsychSignal, AAII, own calculations.

AAII data are published once a week. Youd therefore expect that the near real-time social media data should provide a more timely measure of the market sentiment. As a test, you can look at the correlation between the weekly survey and the lagged daily social media sentiment indicator. If you do that, it turns out that the correlation peaks when the social media sentiment is lagged by 4 or 5 days. This makes eminent sense, given that theres 5 trading days in a week (barring holidays). As expected, the social media holds a clear and predictable timing advantage over the survey data.

Chart 3. The correlation between the aggregated and lagged social media sentiment for S&P 500 and the AAII survey results as a function of the lag. Sources: PsychSignal, AAII, own calculations.

As a technical aside, I dont think we should attach too much importance to the magnitude of the correlations. The correlation coefficients are bound to be high when level variables are used, even if the correlation of daily or weekly changes were low. In this analysis, Im more interested in the pattern and timing of the correlation structure, and those are more evident in the level data.

Compared to the investor surveys, social media sentiment is noisy. You could smooth the data to reduce the noise, but do you really want to do that? Smoothing will necessarily introduce a time delay, and the timeliness in addition to the greater granularity is one the potential virtues of the big sentiment data. But if you go ahead anyway and run a daily moving average through the social media indicator, youll find out that theres a better match between the data sources when you widen the smoothing window. Chart 4 shows that the peak is at about 8 days.

Chart 4. The correlation between the aggregated social media sentiment for S&P 500 and the AAII survey results as a function of the smoothing window. Sources: PsychSignal, AAII, own calculations.

In the end of the day, there are two conflicting considerations here: noise reduction vs. timeliness. What you want to do depends on the use case. If you are thinking of using the social media in a lower frequency model, go ahead and smooth the sentiment data by all means; if you are contemplating intraday strategies, the raw data are likely to be of better service.

The social media data fluctuate more frequently than any survey by construction. But if you were to sample the investor sentiment at the survey frequency, how would the two compare then? This is not a question that can be readily answered with the usual tools of time series analysis. There is no hidden order in the sentiment, and there are no regular cycles or periodicities that can be identified either in the time domain or in the frequency domain.

However, there is some useful data exploration that can be done. It is evident that the investor sentiment has a strong tendency for mean reversion, and there are irregularly spaced peaks and troughs in the data. If we normalise the time series using a suitable transformation, we can simply look at the empirical distribution of the time between the peaks.

Chart 5 shows a dual histogram of the peak-to-peak time for the social media sentiment vs. the AAII survey. Theres two observations to make: (1) There are way fewer peaks in the AAII survey data when compared to the social media, and (2) the peaks in the social media sentiment occur much more frequently; theres often only 10-20 days between successive highs.

Chart 5. The histogram of days between spikes in the aggregated social media sentiment for S&P 500 vs. the AAII survey results. Spikes are defined as periods when the 100-day rolling Z-score of the indicator lies in the 10% right-hand tail. Sources: PsychSignal, AAII, own calculations.

Implied volatility is another well-known gauge of investor sentiment. At any point during the trading hours, it encapsulates the option markets view of the underlying instruments future volatility at a specific investment horizon.

In the U.S. stock market, the VIX index is the most widely followed measure of implied volatility. VIX corresponds to the implied volatility of S&P 500 index options at a 30 days horizon. Similar indices are maintained for many of the major stock market indices globally. Besides equities, implied volatility is being tracked in gold, crude oil, bonds, interest rates, currencies, individual stocks, and generally any instrument or asset class where options are actively traded.

The VIX index is often referred to as a measure of investor fears, and it is also known as the fear index. It is customary to think of VIX levels above 30 or so as a sign of market distress, and levels between 10 to 20 as an indication of relative calm. We should keep in mind, however, that implied volatility is a directionless concept: An expectation of high future volatility does not imply an expectation of falling prices. On the other hand, VIX tends to understate the true volatility at times when volatility is high2.

Implied volatility can be measured over different tenors, and the patterns in short-dated and long-dated volatility can be very different. Furthermore, the implied volatility depends on the option strike, and a comparison of at-the-money implied volatility to in-the-money or out-of-the-money numbers is often informative. This concept gives rise to the option smile, VIX skew, and to indicators such as risk reversals and butterfly spreads. You can also decompose implied volatility into conditional variance of stock returns and the equity variance premium3. And, of course, you can calculate implied volatility based on either put or call prices, and divergencies between the two can be illuminating. You can also have a look at the ratio of put vs. call volume, and so forth.

There is empirical evidence for the hypothesis that option market activity is affected by investor sentiment. In particular, the equity index option volatility smile tends to become steeper when the sentiment becomes more bearish, and flatter when sentiment turns bullish4. There is similar evidence from the Eurodollar options market5.

Such considerations serve to remind us that option markets provide useful information beyond a single implied volatility number. In this post, well keep things simple. What can we say about the relationship between the VIX index and the aggregate social media sentiment for the S&P 500?

Chart 6 shows the social media sentiment against the VIX index, with the sentiment rescaled to roughly match VIX in this time period. It is evident that peaks in VIX tend to coincide with lows in the sentiment.

Chart 6. S&P 500 index (blue line) vs. the aggregated social media sentiment for S&P 500 (orange line, rescaled) and the VIX index (red line). Sources: Quandl, PsychSignal.

Although the social media sentiment seems to swing back and forth more frequently than VIX, there isnt really that much difference between the two (see Chart 7). The distribution and the number of spikes is comparable. In both cases, theres typically not much more than 20 days between successive sentiment highs (or volatility lows).

Chart 7.The histogram of days between spikes in the aggregated social media sentiment for S&P 500 vs. the VIX index. Spikes are defined as periods when the 100-day rolling Z-score of the indicator lies in the 10% right-hand tail (for sentiment) or the 10% left-hand tail (for VIX). Sources: Quandl, PsychSignal, own calculations.

If we look at a scatter plot of the data (Chart 8), it does appear that the social media sentiment varies inversely with the VIX index, with the correlation of about -0.45. When the implied volatility is low, the PsychSignal data indicates that the social media sentiment runs high. And when the implied volatility is elevated, the social media sentiment is depressed.

Chart 8. The aggregated social media sentiment for S&P 500 (x-axis) vs. the VIX index (y-axis) since September 2009 through the middle of February 2015. Sources: PsychSignal, own calculations.

This is rather obvious; theres not much news in saying that VIX tends to be high when investors are feeling low. Interestingly enough, there is a marked difference in the relation if we look at the bullish and bearish sentiment separately. As Chart 9 illustrates, there is a strong inverse relationship with bullish sentiment and the VIX (correlation = -0.53), and this is exactly what one would expect. What is remarkable is the fact that this relationship mostly vanishes in the case of the bearish sentiment (correlation = -0.13).

Chart 9. The aggregated social media sentiment for S&P 500 (x-axis) vs. the VIX index (y-axis) since September 2009 through the middle of February 2015. The left panel corresponds to the bullish S&P 500 sentiment, and the right panel to the bearish S&P 500 sentiment. Sources: PsychSignal, own calculations.

What should we make of this finding? I dont think I have the final answer, but there is one consideration that may be of relevance. As weve seenbefore, the bearish sentiment seems to be less influenced by the history of stock market performance. The asymmetry with regard to VIX reinforces the distinction between market reaction to the bullish and bearish investor sentiment.

Lastly, lets do a brief foray into the stock market sentiment vs. the consumer sentiment. The latter is polled regularly by different organisations. The survey by the University of Michigan is a widely followed indication of the U.S. household sentiment, and this is the data well use here. Is there any relation between the household sentiment and the investor mood?

Chart 10. S&P 500 index (blue line) vs. the U.S. consumer sentiment as per the University of Michigan survey (final readings, brown line) and the balance of bullish and bearish investor sentiment in the weekly AAII survey (orange line). The 250-day moving average of the AAII survey is also shown (thick orange line). Sources: Quandl, University of Michigan, AAII.

Chart 10 above shows the consumer sentiment against the S&P 500 index and the investor sentiment according to the AAII survey. Although a comparison is somewhat clouded by the different frequency of the data, the chart suggests that there is a somewhat tenuous connection between the two. However, it does appear that both the investor mood and the consumer sentiment are low at or near the major stock market troughs, e.g. after the tech bubble crash post-2000 or in the aftermath of the financial crisis. And  not surprisingly  the sentiment recovers with rising stock prices.

As far as the consumer sentiment and the stock market goes, a longer history may be illuminating. Chart 11 shows the S&P 500 index together with the Michigan consumer sentiment since 1952 (the survey was done only quarterly in the early years). It is apparent that the household sentiment does get depressed when the stock market is in the doldrums. I dont think this is necessarily a matter of any great degree of causality between the two: Most likely the consumer sentiment and the stock prices both react to the state of the real economy, albeit with a different lag.

Chart 11. S&P 500 index (blue line, log scale) vs. the U.S. consumer sentiment as per the University of Michigan survey (final readings, brown line). Sources: Quandl, University of Michigan.

Lets try and recap. What have we learnt about the investor mood as expressed in the social media vs. the other sources?

Frequency. Survey-based measures fluctuate at a lower frequency. They seem to measure a slower-moving component of the investor sentiment, whilst the social media captures the more fickle element of the investor mood. In the social media, the sentiment spikes  i.e. the periods of extreme optimism or pessimism  occur much more frequently, even accounting for the difference in the polling interval.

Timeliness. Social media analytics are available through the day and effectively round the clock. As such, they offer a clear timing advantage over other measures of investor sentiment. Of course, this only makes a difference if you have the investment technology to make use of the real time data (and youre willing to pay for it). Depending on the kind of strategies you entertain, the timeliness can be of significant value. For a long term investor who rarely trades, intraday sentiment readings can be almost always safely ignored.

Overlap. Surveys and the social media provide a somewhat similar view of the aggregate investor sentiment. The alternative measures are by no means identical, though. If you are modelling lower frequency phenomena, Im confident that you are better off with a composite indicator that combines social media analytics with surveys of investor mood, market exposure, the implied volatility, and so forth. As far as the general consumer sentiment goes, however, there is only a tenuous connection with the investor sentiment, apart from the major turning points.

Although weve only scratched the surface here, I think that there is enough evidence to draw some tentative conclusions:

Whilst the sources and the measurement mechanisms differ widely across the different proxies of investor mood, there is a fair degree of congruence in the headline readings.

Of the different measures, the social media and the implied volatility are the most timely ones by construction. The social media sentiment is the most fickle among the various alternatives.

If you are interested in a solid low-frequency gauge of the investor sentiment, an investigation of a composite indicator that combines smoothed social media analytics with other variables will be worth your while.

If you are looking at specific stocks or instruments, the social media is one of few games in town (for the other two, look at the option market indicators and news analytics). This is the direction where richer pickings are likely to be found.

Measuring bubble expectations and investor confidence

, The Journal of Psychology and Financial Markets, Vol. 1, No. 1, pp. 49-60 (2000).

The VIX, the Variance Premium and Stock Market Volatility

, The Review of Financial Studies, Vol. 21, pp. 387-414 (2008).

Investor sentiment and interest rate volatility smile: evidence from Eurodollar options markets

, Review of Quantitative Finance and Accounting, Vol. 43, pp. 367-391 (2014).

Thank you for your blog. I live in Canada and am now writing my PhD thesis about investors heterogeneous belief and try to explain low volatility/ beta effect. I just found your blog here, very informative. I am looking forward to reading your new posts.