Social Media Sentiment Analysis and Trading Strategies

Happy New Year! I recently got the opportunity to start doing some work for Ernest Chan’s team at QTS Capital Management and my first project was a literature review of social media sentiment analysis. The PowerPoint presentation above covers the current academic research on social media sentiment analysis, the trading strategies that incorporate social media sentiment, an analysis of the various providers of sentiment data, and much more! If you have any questions or would like the 50+ pages of notes that accompany the presentation, please contact me at coltonsmith321@gmail.com.

Additionally, over the holidays I got a chance to read both of Mr. Chan’s books, Quantitative Trading: How to Build Your Own Algorithmic Trading Business and Algorithmic Trading: Winning Strategies and Their Rationale. I’d highly recommend reading them if you haven’t! They provided valuable insight into properly approaching backtesting and gave me countless new statistical arbitrage strategies to explore. I’m going to have a lot more time this quarter to work on projects for the blog so expect weekly posts!

Advertisements

MACD + SMI Trend Following and Parameter Optimization

Finally a somewhat profitable strategy to analyze! This post will walk through the development of my MACD + SMI strategy, including my experience with parameter optimization and trailing stops. This strategy began with an interest in the Moving Average Convergence/Divergence oscillator (MACD), which I hadn’t yet explored. Also, since the two previous strategies I analyzed were mean-reversion strategies, I thought it’d be good to try out a trend-following strategy. The MACD uses two trend-following moving averages to create a momentum indicator. I used the standard 12-period fast EMA, 26-period slow EMA, and 9-period signal EMA parameters. Although there are a lot of different signals that traders can look at when using the MACD, I kept it simple and was only interested when the MACD (fast EMA – slow EMA) was above/below the signal line. When the MACD is positive it indicates that the upside momentum is increasing, and vice versa for negative values. I then did some research to see what indicators were combined with the MACD. The two that caught my interest were the Stochastic Momentum Index (SMI) and the Chande Momentum Oscillator (CMO). The SMI compares closing prices to the median of the high/low range of prices over a certain period, which makes it a more refined and sensitive version of the Stochastic Oscillator. The values range between -100 and +100, with values less than -40 indicating a bearish trend and values greater than 40 indicating a bullish trend. Normally the SMI can be used similarly to the RSI and indicate overbought/oversold market conditions, but I wanted to focus on using it as a general trend indicator. The CMO indicator is also a momentum oscillator that can be used to confirm possible trends; my backtests confirmed its viability but I decided to center my focus on the SMI. I initially used the standard SMI threshold values (-40/+40) and backtested across the same 30 ETFs from my last post during 2003-2015.

1

2

This was my first time seeing such a high profit factor and a Sharpe ratio fairly close to 1, but unfortunately, the number of trades was too low. A minimum of 700/800 trades is necessary in this backtest to support strategy performance conclusions, however I was happy to see signs of a profitable strategy. As a note, ATR position sizing was used in every strategy backtest, which nearly doubles their Sharpe Ratio. To increase the number of trades made by this strategy I had a couple of ideas. First, since this strategy only enters long positions I tried playing around with the indicators to see if it was also good for entering short positions. This unfortunately was not a profitable attempt. Second, I thought about exploring a separate shorting strategy that made ~400 trades, and simply putting them together. I decided for the sake of analysis it would be better to stick to one main strategy this time, but this is something I’ll consider in the future. Third and finally, after learning about the ATR position sizing I have wanted to experiment with another risk management tool, trailing stops. I added a 7% trailing stop and got the results below. It sacrificed some profit factor for a better Sharpe ratio, but it also made over 700 trades. Although this seems like artificially increasing the number of trades, I was content with the results for the time being.

3

Next, I wanted to see if I could push that profit factor over 4 by optimizing the parameters, namely the SMI thresholds and trailing stop percentages. I did minor parameter optimization in my first post, but this was my first time doing it on a larger scale. I decided to split my time period in half, 2003-2009 and 2010-2015, optimize on the first period, and then use the second period for an out of sample test. I didn’t expect to see a large difference of performance between the two periods, but I was very wrong. I decided to first optimize the SMI thresholds and found ridiculously large profit factors. From this I concluded that 40 was the best entrance, but that the exit signal definitely had the largest impact. I then chose to further optimize the 40/55 (highest profit factor), 40/30 (highest Sharpe ratio), and 40/40 (arguably the best mix of profit factor and Sharpe ratio to support why they’re the standard thresholds).

Optimization 2003-2009
SMI Enter (+) SMI Close (-) Profit Factor Trades Sharpe Ratio
40 45 14.92 152 1.13
40 50 18.25 127 0.99
40 35 9.25 225 1.24
40 30 8.18 258 1.29
40 55 22.14 105 0.91
40 40 11.29 191 1.24
35 40 10.33 201 1.25
30 40 9.94 210 1.27
45 40 11.12 185 1.22
50 40 10.49 179 1.17

Then I tested the 3 strategies with 5%, 10%, 15%, and 20% trailing stops. Based on their performance, I decided which ones to test on the out of sample period and the whole sample period.

Optimization 2003-2009
SMI Enter (+) SMI Close (-) Trailing Stop Profit Factor Trades Sharpe Ratio Continue?
40 40 5% 4.95 535 1.67 *
40 40 10% 7.49 307 1.51 *
40 40 15% 7.94 248 1.36
40 40 20% 9.22 213 1.33 *
40 55 5% 5.06 528 1.7 *
40 55 10% 8.4 286 1.53 *
40 55 15% 9.45 207 1.4
40 55 20% 10.85 163 1.3 *
40 30 5% 4.97 539 1.7 *
40 30 10% 6.82 335 1.51
40 30 15%
40 30 20%
OOS 2010-2015
SMI Enter (+) SMI Close (-) Trailing Stop Profit Factor Trades Sharpe Ratio Continue?
40 40 5% 2.3 436 0.77 *
40 40 10% 2.23 277 0.53 *
40 40 15%
40 40 20% 2.23 221 0.48
40 55 0.05 2.45 430 0.82 *
40 55 10% 2.84 244 0.68 *
40 55 15%
40 55 20% 3.92 146 0.63
40 30 5% 2.17 447 0.71 *
40 30 10%
40 30 15%
40 30 20%
Whole Period
SMI Enter (+) SMI Close (-) Trailing Stop Profit Factor Trades Sharpe Ratio
40 40 5% 3.38 977 1.23
40 40 10% 3.95 586 1.01
40 40 15%
40 40 20%
40 55 0.05 3.54 963 1.27
40 55 10% 4.65 531 1.09
40 55 15%
40 55 20%
40 30 5% 3.28 993 1.21
40 30 10%
40 30 15%
40 30 20%

I was shocked at how much worse the strategies performed in the out of sample period compared to the optimization period. It was really quite interesting, and I suspect there had to be significantly different market conditions to make these differences apparent for all of the strategies. Across the board the best performing strategy was 40/55, with fairly impressive profit factors and Sharp ratios. The higher closing threshold is probably a result of the general upward trending market. This strategy performed best with a 5% or 10% trailing stop but it made almost double the trades with 5% so I decided to try my original 7% trailing stop. The results are below.

4

5

6

Overall, not a bad strategy, with a profit factor above 4 and a decent Sharpe Ratio. It also handled 2008 very nicely. The buy and hold inter-instrument correlation of the 30 ETFs is 0.71 and this strategy cuts it in half. This means it is a well diversified and risk managed portfolio strategy.

7

I’m going to continue to search for profitable strategies and maybe look for a short-term, aggressive strategy to analyze next. I also want to use other metrics, such as the Calmar or Information ratio instead of the Sharpe ratio to get a bigger picture of a strategy’s risk management. Additionally, I hope to apply R’s extensive machine learning capabilities to a strategy in the near future. Thanks for reading!

Acknowledgements: Thank you to Ilya Kipnis and Ernest Chan for their continual help.

Full code:

detach("package:dplyr", unload=TRUE)
require(quantstrat)
require(IKTrading)
require(DSTrading)
require(knitr)
require(PerformanceAnalytics)

# Full test
initDate="1990-01-01"
from="2003-01-01"
to="2015-12-31"

# Optimization set
# initDate="1990-01-01"
# from="2003-01-01"
# to="2009-12-31"

# OOS test
# initDate="1990-01-01"
# from="2010-01-01"
# to="2015-12-31"

#to rerun the strategy, rerun everything below this line
source("demoData.R") #contains all of the data-related boilerplate.

#trade sizing and initial equity settings
tradeSize <- 10000
initEq <- tradeSize*length(symbols)

strategy.st <- portfolio.st <- account.st <- "TVI_osATR"
rm.strat(portfolio.st)
rm.strat(strategy.st)
initPortf(portfolio.st, symbols=symbols, initDate=initDate, currency='USD')
initAcct(account.st, portfolios=portfolio.st, initDate=initDate, currency='USD',initEq=initEq)
initOrders(portfolio.st, initDate=initDate)
strategy(strategy.st, store=TRUE)

#parameters (trigger lag unchanged, defaulted at 1)
period = 20
pctATR = .02 #control risk with this parameter
trailingStopPercent = 0.07

#INDICATORS
add.indicator(strategy = strategy.st,
name = "MACD",
arguments = list(x = quote(Cl(mktdata)),
nFast = 12, nSlow = 26, nSig = 9,
maType = "EMA", percent = TRUE),
label = "MACD")

add.indicator(strategy = strategy.st,
name = "SMI",
arguments = list(HLC = quote(HLC(mktdata)), n = 13,
nFast = 2, nSlow = 25, nSig = 9,
maType = "EMA", bounded = TRUE),
label = "SMI")

add.indicator(strategy.st, name="lagATR",
arguments=list(HLC=quote(HLC(mktdata)), n=period),
label="atrX")

#SIGNALS
add.signal(strategy = strategy.st,
name="sigFormula",
arguments = list(label = "closeLong",
formula = "(macd.MACD < signal.MACD & SMI.SMI < -55)",
cross = TRUE),
label = "closeLong")

add.signal(strategy = strategy.st,
name="sigFormula",
arguments = list(label = "buyLong",
formula = "(macd.MACD > signal.MACD & SMI.SMI > 40)",
cross = TRUE),
label = "buyLong")

#RULES

add.rule(strategy = strategy.st,
name = "ruleSignal",
arguments = list(sigcol = "buyLong",
sigval = TRUE,
ordertype = "market",
orderside = "long",
replace=FALSE, prefer="Open", osFUN=osDollarATR,
tradeSize=tradeSize, pctATR=pctATR, atrMod="X"),
type="enter", path.dep=TRUE, label = "LE")

add.rule(strategy = strategy.st,
name = "ruleSignal",
arguments = list(sigcol = "buyLong",
sigval = TRUE,
replace = FALSE,
orderside = "long",
ordertype = "stoptrailing",
tmult = TRUE,
threshold = quote(trailingStopPercent),
orderqty = "all",
orderset = "ocolong"),
type = "chain",
parent = "LE",
label = "StopTrailingLong",
enabled = FALSE)

add.rule(strategy = strategy.st,
name = "ruleSignal",
arguments = list(sigcol = "closeLong",
sigval = TRUE,
orderqty = "all",
ordertype = "market",
orderside = "long",
threshold = NULL),
type = "exit")

enable.rule(strategy.st, type = "chain", label = "StopTrailingLong")

#apply strategy
t1 <- Sys.time()
out <- applyStrategy(strategy=strategy.st,portfolios=portfolio.st)
t2 <- Sys.time()
print(t2-t1)

#set up analytics
updatePortf(portfolio.st)
dateRange <- time(getPortfolio(portfolio.st)$summary)[-1]
updateAcct(portfolio.st,dateRange)
updateEndEq(account.st)

#Stats
tStats <- tradeStats(Portfolios = portfolio.st, use="trades", inclZeroDays=FALSE)
tStats[,4:ncol(tStats)] <- round(tStats[,4:ncol(tStats)], 2)
print(data.frame(t(tStats[,-c(1,2)])))

#Averages
(aggPF <- sum(tStats$Gross.Profits)/-sum(tStats$Gross.Losses))
(aggCorrect <- mean(tStats$Percent.Positive))
(numTrades <- sum(tStats$Num.Trades))
(meanAvgWLR <- mean(tStats$Avg.WinLoss.Ratio))

#portfolio cash PL
portPL <- .blotter$portfolio.TVI_osATR$summary$Net.Trading.PL

#Cash Sharpe
(SharpeRatio.annualized(portPL, geometric=FALSE))

#Individual instrument equity curve
# chart.Posn(portfolio.st, "IYR")

instRets <- PortfReturns(account.st)
portfRets <- xts(rowMeans(instRets)*ncol(instRets), order.by=index(instRets))

cumPortfRets <- cumprod(1+portfRets)
firstNonZeroDay <- index(portfRets)[min(which(portfRets!=0))]
getSymbols("SPY", from=firstNonZeroDay, to="2015-12-31")
SPYrets <- diff(log(Cl(SPY)))[-1]
cumSPYrets <- cumprod(1+SPYrets)
comparison <- cbind(cumPortfRets, cumSPYrets)
colnames(comparison) <- c("strategy", "SPY")
chart.TimeSeries(comparison, legend.loc = "topleft", colorset = c("green","red"))

#Correlations
instCors <- cor(instRets)
diag(instRets) <- NA
corMeans <- rowMeans(instCors, na.rm=TRUE)
names(corMeans) <- gsub(".DailyEndEq", "", names(corMeans))
print(round(corMeans,3))
mean(corMeans)

SMA 200 + RSI 2 w/ATR Position Sizing Strategy Analysis

Hello all! For my second post I decided to analyze an aggressive short-term strategy. While researching, I found a good amount of literature supporting the potential of SMA 200 and RSI 2 strategies, so it seemed like a solid place to begin. I was also reading through previous posts on the Quantstrat Trader blog and ATR position sizing caught my attention so I decided to add it to the strategy (https://quantstrattrader.wordpress.com/2014/06/11/trend-vigor-part-iii-atr-position-sizing-annualized-sharpe-above-1-4-and-why-leverage-is-pointless/). Parts of the code and analysis later in this post was influenced by the Quantstrat Trader blog and Ilya Kipnis himself, so a big thank you to him! Though the tested strategies proved to not be as successful as I had originally hoped, I was able to draw valuable conclusions from the data while improving my backtesting skills.

To start, I analyzed 2 variations of a strategy that used 4 indicators. In my previous post, the Bollinger Bands and RSI strategy used a 14-period RSI indicator. This strategy uses a 2-period RSI, which makes it a much more sensitive momentum indicator. Two simple moving averages, 5 and 200, were used to track short and long term market trends respectively. Finally, a lagging average true range (ATR) indicator, provided by the IKTrading package, was used to monitor market volatility. This allowed me to adjust trade sizes and normalize risk. The first version of the strategy, V1, uses the 200-day SMA to determine the market trend direction (closing price above/below) and the 2-period RSI to identify overbought/oversold situations that could mean revert. It then makes quick exits, hopefully closing a profitable trade on the long or short side, when the mean reversion is done, indicated by the closing price and 5-day SMA relationship. The signals are shown below:

  • Enter long: Close > SMA 200 & RSI 2 < 20
  • Enter short: Close < SMA 200 & RSI 2 > 80
  • Exit long: Close > SMA 5
  • Exit short Close < SMA 5

The second version of the strategy, V2, is similar to V1 except it only enters long side positions and has an additional exit signal based on the 2-period RSI. The signals are shown below:

  • Enter long: Close > SMA 200 & RSI 2 < 20
  • Exit long: Close > SMA 5
  • Exit long: RSI 2 > 80

Both of these strategies were ran with an ATR position sizing function, using a 20-period moving average and risk percentage of 2%. They were backtested on the following ETFs from 2003-2016, and SPY from 2010-2016/2000-2016. The metrics primarily used to analyze the results were profit factor (gross profits/gross losses), percent positive, and annualized Sharpe ratio. Both strategies made about ~10,000 trades across the ETFs 13 year period. Together these metrics give a fairly complete view of the strategy’s profitability and risk factor.

etfs

Strategy V1’s Performance

v1-1
ETFs w/o ATR Position Sizing
v1-2
ETFs w/ ATR Position Sizing
SPY 2010-2016 2000-2016
ATR? No Yes No Yes
Percent Positive 63.5% 62.8% 64.6% 64.5%
Profit Factor 1.59 1.31 1.72 1.44
Annual Sharpe 1.99 1.51 2.27 2.08

Overall, with or without ATR position sizing, this was not a winning strategy as implemented. Most surprisingly, position sizing negatively impacted the profit factor, percent positive, and annualized Sharpe. The strategy did display its ability to adjust to market volatility.

iyrwatr

The above figure shows the strategy’s performance on IYR, the US Real Estate ETF. Looking at the ETF price, position fill (blue columns), and ATR indicator (bottom blue line), you can see the gradual reduction in position sizing leading up to the 2008 recession and the transition to short side positions. The implementation of the ATR position sizing was a definite success in appropriately adjusting the trade sizes based on market volatility which is shown nicely in this figure.

After analyzing the position fill on this ETF as well as others, a common flaw of this strategy seems to be holding long positions too long into a downturn and consequently not entering short positions soon enough. As visible in the figure above, these transitions are quite mistimed. If this strategy could improve these timings I believe it would drastically improve its profitability.

Strategy V2’s Performance

v2-1
ETFs w/o ATR Position Sizing
v2-2
ETFs w/ ATR Position Sizing
SPY 2010-2016 2000-2016
ATR? No Yes No Yes
Percent Positive 60.7% 64.8% 61.4% 63.7%
Profit Factor 1.55 1.48 1.40 1.33
Annual Sharpe 1.59 2.15 1.20 1.66

The results of this strategy told a slightly different story. Across the ETFs and SPY, the profit factor was negatively affected by ATR position sizing, but the Sharpe ratio was improved. Initially I hypothesized that the ATR position sizing should increase the Sharpe ratio since it prevents huge losses by decreasing risk volatility, so this was exciting to see.

ls_iyrwatr

Once again, visible in the above IYR figure above, the strategy does a fairly good job at sitting out severe downtrends. Strategy V1’s trade-side distribution was about 80% long and 20% short so this strategy was essentially just the long trades with the additional exit signal that may or may not have been beneficial.

I also tested these strategies with more sensitive signals by changing the RSI thresholds to 90/10, but that did not improve their performance. Overall, these strategies let me experiment with a short term trading strategy and the metrics used to evaluate them, which will be useful in future projects. With regard to the ATR position sizing, I strongly believe that it would be much more helpful when applied to a long term trading strategy. In the case of a short term trading strategy, the trade sizes become essentially irrelevant as the number of trades increases making the strategy’s percent positive, win-loss ratio, etc. more important. In a long term trading strategy where positions are potentially held for months, the trade size is much more important. Based on the qualitative impact ATR position sizing had on this strategy, I now understand how much of an essential risk management tool it is.

Another tool used to help adjust for risk that I want to incorporate into my next strategy is trailing stops. The figure below is the MAE plot for strategy V1 which can be used to determine an appropriate trailing stop. For example, a trailing stop of 4% added to this strategy would cut a significant portion of the losses.

risk

Now that I have a basic understanding of Quantstrat and evaluating strategy performance, I plan to focus on finding more profitable strategies in my next projects. I still want to dive into something a little more statistically/mathematically dense. Currently I am taking an advanced applied mathematics course, so maybe I’ll get some ideas in there! Additionally, I eventually want to incorporate machine learning into a strategy. Thanks for reading!

Bollinger Bands and RSI Analysis with Quantstrat

Hello, my name is Colton Smith and this is my first hands on experience backtesting quantitative trading strategies. For the last 6 months quantitative trading has been a personal topic of interest that I have dedicated a lot of time researching to.  I’m currently a senior at the University of Washington, majoring in Industrial and Systems Engineering with a minor in Applied Mathematics. Outside of my major, I have taken computer sceince courses and been heavily involved in the UW Math Club which I am now the President of. With this background and experience I have become very familiar with statistics, probability, and R. This upcoming year I will be taking more computer science courses and continuing quant projects to further prepare myself for internship and job oppurtunites.

The most logical place to begin was the Quanstrat package in R. I wanted to start with a simple strategy that would let me explore the different functions of Quanstrat and play around with some data. Being already familiar with Bollinger Bands, a volatility indicator, I thought it would be interesting to analyze them combined with RSI, a momentum indicator.

Therefore, I chose to analyze this strategy from 01/01/2010 to 12/31/2015 because no combination handled 2008 very well and I wanted consistent data to work with. The Bollinger Bands indicator used a simple moving average with n = 20 and RSI used a weighted moving average with n = 14. The buy signal was when the close was below the lower Bollinger Band and the RSI was below the lower threshold which would indicate the stock may be oversold and could mean revert upwards. The sell signal was when the close was above the upper Bollinger Band and the RSI was above the upper threshold which would indiacte the stock may be overbought and could mean revert downwards. I decided to gather data for the 150 different combinations of Bollinger Band standard deviations (1,1.25,1.5,1.75,2,2.25), upper RSI thresholds (60,65,70,75,80), and lower RSI thresholds (20,25,30,35,40). These ranges were chosen to ensure that all combinations executed a decent amount of trades during this period. Some combinations (<5%) still didn’t execute very many trades, which may have skewed some of the data. These strategies were tested on SPY with a starting account equity of $100,000 and each buy signal making a purchase of 500 shares. I chose to analyze the combination’s performance based on its CAGR, Compound Annual Growth Rate. For reference, the SPY’s CAGR in this time period was 10.6%.

1
Figure 1
2
Figure 2

Figure 1 displays the results of each combination. The best performing combination was with Bollinger Bands standard deviations of 1 and the upper/lower RSI thresholds set at 80/40 which had nearly a 40% CAGR. Looking at figure 1 and 2 together you can see how the standard deviation of the strategy’s performance shrinks as larger Bollinger Bands standard deviations are used. This is because when starting at the beginning of figure 1, where lower standard deviations are used, the RSI thresholds are controlling the strategy. However, when the Bollinger Bands standard deviations increase, they dominate the strategy. Additionally, as displayed in figure 2, the convention that using Bollinger Bands with standard deviations of 2 for the best performance, is supported.

As visible from the highlighted combinations in figure 1 and from the table below, strategies with higher upper and lower RSI thresholds performed better because they bought more often and sold less frequently. I believe this worked well because of the upward market trend. The 3d surface plot below helps visualize the influence of the upper and lower RSI thresholds. This plot is with Bollinger Bands standard deviations of 2 and nicely shows how increasing the lower RSI threshold has a drastic effect on the CAGR, much more than the upper RSI threshold does. For reference, the red dots were combinations that didn’t beat the SPY CAGR, the yellow dots were ones that outperformed the SPY CAGR but were below 25%, and the green dots were combinations that had CAGR of above 25%.

2
Figure 3

Overall, this project was very helpful to learn how to work with multiple signals and paramsets in Quanstrat, as well as some data visualization techniques. Next, I plan to explore a strategy where I can utilize my statistical knowledge with one that makes much more frequent trades so I can analyze risk and profit factors.

Acknowledgements: Thank you to Ernest Chan, Ilya Kipnis, and Brian Peterson for answering my elementary questions about Quantstrat and backtesting.