Machine learning in trading strategy?

Question: Does anyone here apply machine learning techinques (such as Neural networks, PCA, local sensitivity hashing) to his trading strategies?

There are two main commercial vendors of AI models for retail traders:
Neuroshell and TradingSolutions.

I think that they are offering a very easy tutorials allowing to grasp the use of those models in trading systems.

On both sites there is a lot of free information about basic models. In trading solution there is even a demo version which gives you several models and gives you the opportunity to make new models.

If I can summarize it in really rough way basically the basis of everything is:

* To select the predictive inputs (mainly technical indicators) and try to predict with them something else output.

* Use the output as a trading signal.

For example:

You have 2 inputs
You use a moving average of period of 20 as input.
You use a second moving average with period 10 as input.

You want to predict the next day close.

So the rule may be if the predicted next day close is higher than the next day open then buy.

If the predicted next day close is lower than the next day open then sell.

You train the model. And you backtest the model over a past unseen data.


So that is the basis and from there start the variations including their use in portfolio.


I have two remarks:

* The problem is that trading with neural networks is not an easy task. My opinion is applying a model over an already viable strategy or idea will allow to squeeze more profit from the idea compared to the rough model.

* Another problem I may say is luck related. Very often you can confuse the optimization with the luck. Very often the results you get are purely due to the luck. Monte Carlo simulations may help a lot.

Please check the book http://www.evidencebasedta.com/ and Monte Carlo
Permutation Evaluation of Trading Systems
http://www.evidencebasedta.com/MonteDoc12.15.06.pdf

I just want to add another remark that by using commercial software within the software is integrated a fool proof.

In Neuroshell for example they are offering to you some default settings that will limit the overfitting. In trading solutions it is the same. If you use your own model or an open source software and you do everything by yourself you need to be carefull what are you doing, for example adding too many weights to the model and too many neurons and hidden neurons will increase the risk to overfit the data and not to learn from the data.

Get real-time feed of live data into Excel

I have found this list of sources of data for real time feeds of live data. I think it may be useful that is why I share the list here.  

Get real-time feed of live data into Excel
Get free historical data into Excel, Matlab or Java
Foreign Exchange, Forex, FX

www.forexite.com (Forex, 1 minute ticks), CSV download
ratedata.gaincapital.com (Forex, tick data), CSV download
www.oanda.com (Forex, daily), CSV download
www.truefx.com (Forex tick data) after login
www.dukascopy.com (Forex, ticks), CSV
Debt/Interest Rates

historical time series
www.euribor-ebf.eu (Interest-Rates: “Euribor, Eonia”), Excel download
www.databank.rbs.com (Exchange Rate, Interest Rate, Consumer Price Inflation (CPI), Commodities: “Brent Spot, Henry Hub Spot, I P E Brent crude futures 1-pos, I P E Natural Gas Index, OPEC Fixing, US EIA Crude Imports, WTI Daily”), Excel download
www.ecb.int Interest-Rates and Government Bonds, web tables and partly Excel download
(German) www.boerse-duesseldorf.de (Bonds, daily), Website table only
(German )www.bundesbank.de German money and interest-rate statistics
current
(German) www.baadermarkets.de (Bonds)
(German) www.westlbmarkets.net (2012-07-12: Just went offline, hopefully online again soon), Swaps, Caps, Floors, CSV download
Stock / Indices

historical time series
finance.yahoo.com/(Stocks, Indices), Excel download
www.google.com/finance, Website table only
www.nasdaq.com (Nasdaq), Excel download
mba.tuck.dartmouth.edu (portfolio data), text download
(German) finanzen.handelsblatt.com
www.investopedia.com, Website table only
www.nasdaq.com, Website table only, intraday, daily
www.standardandpoors.com (S&P 500, daily, yearly)
current
finance.yahoo.com (Stocks, Indices)
www.bloomberg.com (Stocks, Indices)
Energy

EU http://www.epexspot.com/de/marktdaten/intraday-handel (German), Website table only
US http://www.gdfsuezenergyresources.com/index.php?id=33 CSV download
General Historical

(German) www.onvista.de (Stocks, Indices, Warrants, ETFs …, daily Quotes) Website
www.indexmundi.com World-wide econometric data and commodities
www.measuringworth.com US, UK, Japan, China with interest rates, GDP, inflation, gold and silver prices, cool long term, Website table only
Sites with Summaries

scraperwiki.com (The community collects free data from the web and converts it into CSV, etc.)
wikiposit (The author collects free data from the web and converts it into CSV, etc.)
Matlab Trendy (The community collects free data from the web and converts it into plots)
Economic data

World

www.indexmundi.com World-wide econometric data
OECD.StatExtracts includes data and metadata for OECD countries and selected non-member economies.
Google Data e.g. Market Capitalization in GDP: Google Data on Marcet Cap in % of GDP
European Union
Eurostat
www.ecb.int
www.destatis.de Statistisches Bundesamt
United Kingdom
www.statistics.gov.uk
United States
Federal Reserve Economic Data – FRED
www.census.gov
www.bls.gov
www.ssa.gov
www.treasury.gov
www.sec.gov
www.economagic.com (Web, Excel download after login)
www.standardandpoors.com/indices/sp-case-shiller-home-price-indices (Shiller-House Price Index)
www.data.gov Public data of all US government organizations
Live Cams

www.boerse-frankfurt.de (German Dax)
Buy Data:

(I did not try any of these, but they look interesting.)

www.cmegroup.com (derivative prices)
www.ivolatility.com (Implied volatilities)
commodities.euronext.if5.com (Cocoa Future, Robusta Coffee Future, White Sugar Future), 15 min delay, XML
www.eoddata.com (End-of-Day Stock), Excel download and more
www.historicaloptiondata.com (Option price data)
www.livevol.com (Implied volatilities)

Trend following EA performance results

I will post some recent performance tables of Brain Trend EA on its default settings og lenght 50. The results show clearly that it performed well on 30m, 1 h and 4 h time frame.

On 15 m time frame it did not perform on any settings.

I somewhat missed the performance because I was focused exclusively on the 15 m time frame and I knew it was not working there. So I am glad with the results because the idea here is not to produce EAs and hoping to find the holy grail by chance ;).

The brain trend is a sound system with its limitations.



От EA tests




От EA tests




I will add the results on 1 h time frame and the 4 hour time frame with the same parameters of lenght of 50.

A you can look the results it is typical for trend following system: the % profitable trades is below 50 % but the average profit trade is bigger than the average loss trade.

As you can see 43 % is very good performance for trend following system.

Also the largest gain (77.4) has to be much bigger than the largest loss (-41.10). However you need to see if all the good results are not due to just one trade.

Also it is important to have a look at the consecutive losses. This is also a critical parameter for trend following trading systems.

In the case they are 6. This is a lot but hopefully the consecutive loss (-132.27) is not very high compared to the biggest profit trade (230.39).

What is a bittle bit surprising of this system is the consecutive profit of wins of 4 (398.89). That is surprising for a trend following system. In fact the brain trend is an upgraded trend following system, I would say it is an impulse following system. When the market state get more predictable a persistent it is able to follow it. Check about the market states analysis.

Well this is not much, but this is what we get from Metatrader as statistics.

Now I will add the screen shots:



От EA tests




От EA tests


So you can argue that the data is not big. And yes it is true. The performance of the system is closely related with the predictability of the current market state. I was thinking about a seasonal pattern but I do not find any clue about it.

The discussion and the download files are here.

Forex Remote Viewing

Can you apply remote viewing for Forex trading? Theoretically Remote viewing can be applied everywhere. Of course it is obvious that the tools will be used to suck a lucrative activity as trading. Historically the first methodologies were developped for the Stock market.

And guess what if you google search those key words that would mean that you had the idea that the remote viewing techniques may be useful for trading. And if you had the idea that means very often that somebody else had the same idea too. Here I will give you a link to a you tube promotion video that matches what you are looking for.

However I will bring the discussion a little bit further.

The video and the course is from Ed Dames. Who is Ed Dames and how is he related with remote viewing.

Major Ed Dames is a distinguished military intelligence officer from United States Army (ret.).

After the declassification of the Project Stargate many of the military personnal went into the private sector.That explains how this and other courses are created.

So with Ed Dames you will get first hand experience in remote viewing. As this is very technical stuff the details are what matters most. So if you are going to put real money into it be serious and learn from the best.

Here I end with the recommendations and I will show the limitations of the method. Check the previous post about the risk.

The problem is related with the multiple time lines. There is a theory that there are multiple time lines and if you get a result you can't be sure that the result is withing your time line.

For more explications you can check this video from Courntey
Brown: 



So who is Courntey Brown? He has a  Ph.D. degree from Washington University (St. Louis) in 1982 in political science with an emphasis on mathematical modeling and he is a leading scholar on the subject of "remote viewing" and he is representing the non profit public research with the Farsight Institute.

So he is another authority in the field if Ed Dames is former military agent and turned toward the practical use and teaching, Courtney Brown is theoretical and scientific.

The discussion is here:

The Stanford Discovery and the remote viewing

The Stanford institute were trying to understand what makes the psychics succesful in their predictions and what makes them damn wrong. They tried everything for spi purpouses even out of body experiences.

As I am from the french school of though and we disnguish in everything two twings all the time I can say that there are two ideas.

The firt way is ESP (extra sensory perception) is a a particular state of mind ans special mind conditions are necessary in order to be a psychic. And you will find none of this in those documents, even if Ingo Schwan uses an alternate mind states.

However in those condition you really cannot understand what is real and what is imagination. On the other hand you are very vulnerable for psychic attacks. Some magick practitionors know that very well.

The other theory is that no particular state of mind is necessary. We are constantly in the flow of information. They call that the matrix, long before the film. What is needed is a process to get the information before the creative imagination take over. They call all picture that is coming in the imagination an Analytical overlay and is most of the time rubish. So you will not find any special meditation techniques. Why? Because there arent any special mind techniques. The only thing you can do is to take bit per bit information from the matrix and record it on paper. And you make more and more iterations. And you process the information before the imagination takes over. There is the secret and that is a BIG discovery and most of the people interested in the new wave are completely unaware if it and they are a simple pray of their imagination and not so friendly non material influences. As traders are particularly vulnerable it is not a good idea to experiment with those kind of stuff, especially if you are not under supervision and control. Anyway it is dangerous.

The truth is ()just an opinion LOL) that we are in a big ocean of information. There is a a barrier between it and us. Any force attack of the barrier is protected (the creative imagination under alternate state if mind). That is why other methods have been developped taking information bit per bit, per bit, hacking it, this is what remote viewing is about.

Have you ever solved a problem in less than 4 seconds? If you did that is probably because you had an access to that source of information.

Read more:

Trading strategies: using computing simulation to maximize profits and to control risk

Download link: 

This is a free e-book from Larry C. Sanders from his website tradelabstrategies.com. On this discussion forum you can read an interesting discussion about it.

Larry C. Sanders addresses the topics of probability, marble game and MonteCarlo simulation. He has also designed a software program TradeSim which he sells at his website. The book can be downloaded at this site for free (this is the bookmark link). This book also talks about the Monte Carlo analysis of trades.

Monte Carlo simulation for trading system's risk control for real newbies

"What is the last thing you do before you climb on a ladder? You shake it. And that is a Monte Carlo simulation" 
Sam Savage, Stanford University

So if we make an analogy the ladder is the result of the trading system's back tests.

Have a look at the picture. Here we have 10 trades of our imaginary trading system, they are numbered from 1 to 10. Each trade is either good or bad, what is important is that we have an equity curve that is formed by those trades.



Here you can imagine we have a result consisting of 10 trades. What we do is that we shaken those results randomly, however no trade is added or deleted. What we do is that we just replace their position randomly. This is called selection without replacement. In other words we change the positions of the trade sequence of the system. 

By doing selection without replacement we duplicate the probability distribution of the initial trade sequence.

If we apply this many times on an initial trading sequence we will get something that looks like that, check the screen shot.

The beginning and the end is at the same point because it is the same trading sequence, but the paths are different.

However the next questions arise, how many permutations we can do? 10, 100, 10000 or 100000.

The practical limits are given by the statistics.

Imagine we have 1000 trades. By calculating the permutations that will give us:

1000*999*998*997*...3*2*1

The number we get is unbelievably big. The number of permutations of 200 is bigger than the number of all atoms that exist in the whole universe. So it if obvious we are not going to do all those permutations.
Hopefully there is a simple algorithm that will help us:


If the probability p is close to 0.5 So the 95 % confidence interval reduces to the following formula:

If we have a flipping coin of 1000 times and we want a confidence level of 95 %. The confidence level will explain how likely the 500 heads is placed within the confidence interval around the estimation.


For the Gaussian distribution we can be 95 % sure that the value is accurate within 3.1 %. With 95 % probability the coin will will fall between 469 and 531 times a head from the 1000 trials you can do.


If we use the formula we get

0.5+1/sqrt(1000)= 0.5+ 0.3162277

0.5-1/sqrt(1000)=0.5- 0.3162277


So we can share that the value is 95 % sure that is accurate within 3.1%.

Practical use of Monte Carlo simulation for traders

So what does it means in practice. In fact the most important aspect of this kind of Monte Carlo analysis is the analysis of the drawdowns. For example with a given confidence level you can say what that worse drowdown can be.

Look again at the shot where are the equity curves. We are going to be interested at the worse case scenario.

Imagine the original system has 500 USD drow down an 1500 USD average profit per year. In our hypothetic example that would be the system with the blue line.

So now we are going to look at the hypothetical equity line with the worst drow down.

If your Monte Carlo analysis show you for example 1000 USD as the worse drowdawn at 95 % confidence level that means that there is 5 % chance of facing 1000 USD drow downs before making any profits. Well if you look at even lower confidence levels the expected drowdown can be lower.


However the worst drow down is not more than the average profit of the system. According to Urban Jaeckle and Emilio Tomasini it is unacceptable to have a drowdown lower than the average profit per year.


The main limitation of the Monte Carlo analysis are that if your initial results are curve – fitted your results will be a nonsense. The Monte Carlo analysis is good only when it is applied for a sound trading system and not to an over-fitted one.

If your trading system is sound or not that is just another thing. The Monte Carlo simulation need to be applied with care but it is a valuable tool for risk control of a sound trading system.


Recommended readings 


Urban Jaeckle, Emilio Tomasini, 'Trading Systems A new approach to system development and portoflio optimization' Hariman house, p.101-108

Farell, Christopher, 'Monte Carlo models simulate all kinds of scenarios' Business Week 2001

Discrete Event Simulation a First course at this link

Range and timing: John Last's Forex Jokers

John Last's Jokers are useful in three games on the market ;) (market states).

For each game (phase) we have one joker.

You can play three games and they come over and over again. Range, Breakout, Trend, Range, Breakout, Trend and so on.

For each game you have one joker ;)

This is a tactic for range daytrading

Here look at this screen shot below.

Here I try to make an accurate timing of the range.

The idea is as follows. Range can be played but we need it to be fairly broad. This is clearly visible on the fractal bands, but also the common Bollinger is also useful.

In the narrow range (from the perspective of fractal technical analysis) is difficult to be played. Why? Because then orders accumulate  on both sides and it is a matter of chance where will break, and the fake break - outs are pretty nasty. We have to wait patiently for a breakthrough, not to bet which way will break. Well the general technical analysis bet for the continuation of the previous trend but on the 5 m time frame this is ... hard.

OK loot at the screen shot again. We have a christmass tree here but if you wait for Santa Claus this is a good thing to do.

1. Brain trend setting 14 (You can optimize the smoothing visually or run the Brain Trend EA in order to get the best performing set on the current market state)

2. Bollinger bands with 20. This is the fast Bollinger.

3. Bollinger bands with 60 period. This is the slow Bollinger.

The idea is to match roughly with Bollinger of 20 on  15 m time grame. So I multiply the period of three (According to Bollinger should do 2.1 standard deviation increase in the period but you need a special indicator to do that).

4. Oscillator-based digital filter and here a simple smoothed stochastic works (on this chart I use a commercial product but nowadays you can find good indicators easily)


Watch for signals from reaching the two borders of Bollingers the fast and the slow. You can use only the slow if you want.

Then use your price action skills, or use oscillators or wait for convirmation from Brain trend.

Even better you can have your individual oscillator based range system.

Or look for overbought or oversold  levels of your favourite oscillator.

What makes this different is that we do not identify a fractal break - out from the FGDI, or iVAR indicator. And not peak in the indicator: variation of the Hurst exponent.

A can add many confirmations of stochastic on the 15 m time frame but it is useless.

The idea here is that we have two independant statistical tools that come together with pattern recognition.

The Bollinger bands are giving a deviation from the central tendancy.
The Fractal indicators need to detect an anti-persistent situation.

Then you can bet for the reversal.


About Money Management seems reasonable following system:

When signal meets the criteria found three items:

1 Close for a small profit to cover the transaction costs
2 Close at the reaching the midline of the 60 period Bollinger
3 When you reach the opposite levels of resistance and support levels of the channel opposite Bollinger.

Position 2 and 3 can have dynamic stf opso Brain The line.



-Be careful with the News



Trend Joker

Never  counter the trend by having  fractal dimension in the RED of FGDI and the fractal bands have to be wildly apart there. Again a brutal reversals are possible but to catch them you should not suppose it but to react ASCTrend or Brain trend are a good choice.



Range Joker 2

Rage divide it into two types. Narrow and wide. Narrow is risky.


Break-out Joker 3

We have a valid break when fractal dimension crosses 1.5. If this is accompanied by a large variation of the Hurst exponent is morethan- a strong signal.

Again Joker # 1.

Entropy Scalper EA with AI

You can download the Entropy scalper with chaotic kernel.

The download is available only for registered and logged users.

The performance can be compared.

The idea:

The idea is to detect as for the fractal spcalper a pocket of predictability. Within this pocket of predictability we are taking a directionnal signal based on the current volatility.

So there are thre main components:

a. The pocket of predictability.

The pocket of predictability is a situation when we have an increased predictability. We do not know why this occurs, we are interested in the fact. And this pocket can be discovered using different tools. In metatrader we have fractal dimension indicators family: FDI,FGDI, iVAR, Hurst exponent, and the fractal scalpers are using then.

Here the entropy scalper is using an entropy indicator. The entropy is another way to measure those mystical pockets of predictability. Here we look for a drop of entropy below certain levels. The default levels are 0.6.

b. Directionnal signal

The directionnal signal is using a special set of indicators. The indicator used is the Entropy math indicator. The Entropy math indicator is a whole story by itsels. This is not a traditional indicator but and it comes from the work of John Conover, you can visit his site for more details. The entropy math is a Metatrader version of his work, the article is called quantitative analysis of high entropy systems.

However we use a set of four Entropymath indicators as inputs and we use a Chaotic kernel in order to optimize the values with the genetic optimizer. Se keeping it simple the Entropy math is an input that is used by a "neural network" (The chaotic kernel with the genetic algorythms is different but let keep it simple.)

c. Market State:

The market state is used trough the stop loss, take profit and live hours settings.

It is obvious that this is a critical component. We use the genetic algorythm to find those values. However it si advised to use your own view of the market and additional tools as the Lyapunov exponent.

And of course take care of the news releases.



Installation:

In the archive there is an EA files and two indicators. If you do not have those indicators you need to install them in order to run the EA

Some idea:

As you can see there are many parameters that are common among all the fractal scalper indicators.

You can check the fractal scalper description.

One of the most important parameters concern the stop loss the take profit and the live hours parameters.

Those parameters are specific for the current market conditions. As I recommend that you use open prices only for optimization (otherwise it will takes a lot of time), you can't use stops below 15 pips.

There are the paprameters if the two chaotic kernels.

You can optimize both or one of them.

x from 0 to 1 with step size 0.01, 0.001 or 0.0001

limit map from 1 to 1000 with step size 1

r from 1 to 4 with step size 0.01, 0.001

EA Gann High Low Activator

The work of Gann is indeed very interesting it is the result of someone who has dedicated his entire life to the markets.

Howevever what we know about him today is not everything he has created. Some of his works never quited the manuscript stage.

I think that he was using everything he can, all the available technology of his day in order to read the markets. I think he was a visionary. Many can't understand but the simple bar charts of today were at the time something completely astonishing.

Yes today we have the summum of all the technical analysis available on internet, most of the readers here have gigabites of technical books and works in their hard - drives.

But on the other hand if Gann had an odge during his lifespan markets over the others the same is not true for us.

The market is something very specific, not all markets are the some and the markets evolve during time. I mean that the most computarized markets of today are fundamentally different from the markets in which Gann has practices his methods.



So there are two difficulties.

1. The Gann works and methods are incomplete (many of his works are not available).

2. Even if they were they were created for markets totally different from the market of today. That means that authomatic implementation is not going to work.



And yes, what can be done is to be fimiliriased with his methods and to have a creative approach.

Some even say that Gann was one of the first guys who actually used algorythmic approach. It is about the Gann high - low activator (now by the way entirely coded in mql and freely available to everybody, in its time that was the edge of the research).

Have a loot at: Krausz: W.D. Gann Treasure discovered



I made my own EA Gann High Low Activator. It is based on the indicator Gann High Low activator SSL from the code base.

You can download the indicator with the EA from here, just click on the download button. You need to install the indicator in order to make it work the EA.

And here the good news are over. Yes I made some testing but I did not find this EA strategy reliable on the recent market of Euro/Dollar. You can make your own tests and share.

Parameters:

The EA is based on the moving average generic EA, that means it shares the generic money management module. However in order to make it work I changed the trading rules a little bit (I do not pretend that I got it right).

So there is only one parameter and it comes from the High Low activator it is the Lb parameter and that is simply the number of bars you use for the computation of the indicator.

The rules can be interpreted as the basic single average system towards the price. Check the picture and you will understand.

If the Close is < the Gann High Low Activator indicator we Sell

If the Close is > the Gann High Low Activator indicator we Buy

DOWNLOAD the EA from here