Curve fitting or optimization: Statistical versus Phenomenological optimization

When we try to optimize using an optimization algorithm are we doing curve fitting and our model is it going to work in the future?

This is a major problem, there are authors that suggest when we test our system for a long period we should not change its parameters anymore. Others prefer to optimize everyday the parameter of the system.

Where is the truth?

I will try to see the things under other perspective. When we try to optimize a system, we are doing basically a statistical optimization. We optimize return on account, maximum winners, Profit/Loss ratio etc. And we optimize statistically with an algorithm the system.

So how robust are the results when they face the real markets. Well, this is tough question.

Sometimes it works, sometimes it does not. That is the reality.

On the other hand not testing a strategy is suicidal. Some authors state that they limit the variables of the system so the over-fitting is less probable concentrating on the most important things like trend direction and support and resistance zones.

Statistical versus Phenomenological optimization

Phenomenological Theory. A theory which expresses mathematically the results of observed phenomena without paying detailed attention to their fundamental significance

With this kind of optimization we choose to have not one system for all market conditions but some systems phenomenologically optimized for a certain condition.

And we need some criteria to switch between the systems. The Fractal dimension graph index is an useful tool in the phenomenological approach.

For example we can use one system when we have antipersitent fractal characteristics and another when the characteristics of the price are persistent. In fact the adaptive robots are a hype, when they know when to stop trading a particular strategy and when they start again.

Chart patterns or Chaos attractors?

All the chart patterns can be analyzed as attractors.

That is a modern explanation of the chart patterns that make sense. I will give a definition of an attractor using the Wikipedia. The problem is that the traders do not understand the science of chaos, and the scientists do not understand the trading. As for me I do not understand them both but I will try my best.

An attractor is a set towards which dynamical evolves over time. That is, points that get close enough to the attractor remain close even if slightly disturbed. Geometrically, an attractor can be a point, a curve, a manifold, or even a complicated set with a fractal structure known as a strange attractor. Describing the attractors of chaotic dynamical systems has been one of the achievements of chaos theory.

Please read this short article in the Wikipedia

http://en.wikipedia.org/wiki/Attractor

What is important is that "A dynamical system is generally described by one or more differential or difference equations. The equations of a given dynamic system specify its behavior over any given short period of time. To determine the system's behavior for a longer period, it is necessary to integrate the equations, either through analytical means or through iteration, often with the aid of computers."

And The Fibonacci numbers are defined using the linear recurrence relation. And they can be helfull in the analysis.

According to my analysis the chart patterns can be analyzed as chaotic attractors.

Fixed points: In a trend environment, Higher highs, or lower lows, does it look familiar

Limit cycle: Oscillating price patterns: In fact all the technical analysis chart patterns are in this category: Triangles, Wedges, Harmonic patterns etc.

Limit tori: Complex patterns. It is arguable if they are a part of the technical analysis.

maybe the Elliott Wave Sequence may be close, but I am not sure.

Strange attractors: are not part of the technical analysis. Of course the Elliott wave theoreticians claim that they can predict a lot but their instruments are not adapted to the this task. Moreover those attractors are not necessary for trading, it is better to use lower order attractors for trading.

What is important that the price in the Forex markets are not in the void. They are in a phase space.

http://en.wikipedia.org/wiki/Phase_space

The phase space is a space in which all possible states of a system are represented, with each possible state of the system corresponding to one unique point in the phase space. For mechanical systems, the phase space usually consists of all possible values of position and momentum variables.

For example the daily volatility can be analyzed in practice as a phase space of the price time series for the day. Of course as according to the hypothesis that the distribution is not normal but stable paretian with infinite variance this does not holds true, but is an useful practical approach (in fact we can detect in real time with the peaks of Hurst difference how and when the a powerful shift occurs).

Sensitivity to initial conditions

Sensitivity to initial conditions means that each point in such a system is arbitrarily closely approximated by other points with significantly different future trajectories. Thus, an arbitrarily small perturbation of the current trajectory may lead to significantly different future behavior.

In practice a powerful spike even it is corrected will influence the future price action and will set a new set of solutions. If a spike modifies the current phase space (current volatility) it is a signal that the system will be perturbed.

In practice we can analyze the beginning and the end of the market periods.

Let look what happened in the Euro.

For example when we had a powerful trend in September that state was set be the initial conditions. Unless they were not perturbed they set particular attractors that were working. In this case the trend was so simple that it was unbelievable how simple trading can be. But some people did not make money because they cannot believe that and countered the trend for a reversal.

After that Happened in October a break - out in the European session that was directed downwards. What did that mean? That means that the structure of the market has been perturbed. The market started to make another type of patterns: oscillating patterns.

In November this week a powerful shift and spike has occurred. This spike perturbed the structure of the chaotic attractors (cyclic type of patters with nice swings ). The subsequent break - out upwards was a part of the new structure and will participate in the new chaotic attractor that will emerge.

So we can analyze the price action as chaotic attractors in a particular phase space. What is important to know when a powerful shift in the structure occurs. We cannot know how, when, and why and that cannot be predicted. You can analyze that by your experience or you can use an algorithm (like this which is posted on this blog).

The most common errors that is made by the technicians is when a powerful shift in the structure occurs they try to use a pattern that was in force before.

That is a point when the complex neural net models fail, because they are used for a structure that is not anymore valid and a new structure is about to emerge.

This is just a theory guys and gals. But the calculation of the Hurst exponent showed a clear long term process in the markets. The lyapunov exponents calculations showed that even sometimes the processes are really not dissipative.

The phase of the digital filter

Here I add some shots to illustrate how the phase of a digital filter can make a difference in the trading signal.

The first shot has a phase of the signal +100

The second shot has a phase of the signal +5 (neutral)

The third shot has a phase of the signal -100

The degree of smoothing is 6.

http://www.mediafire.com/?uj5fstmoqynlxgg

Fringe Technical Analysis

I am a fan of Fringe. So that is why I can call those ideas as Fringe Technical Analysis. Fractal technical analysis does not sound good anyway.

So the fringe technical analysis covers all the unconventional and strange ideas and methods.

According to the dictionary it means:

something that is marginal, additional, or secondary to some activity, process, or subject

In the blog as a link has been developed some day trading strategies who do not depend on the classical notions of trend. Hey we do not need that stuff. What is important is if the price movement is persistent or anti-persistent and the volatility.

Pattern recognition?

Does the pattern recognition work?

This is really tough question. The technical analysis rely heavily on pattern recognition. Here I will share my opinion without covering all the issue.

I prefer to rely on more marginal patterns that the most used. I mean that when we have a formed typical pattern, everybody sees it. And when that happens a very complex relationship develops between the market participants. In the litterature it is referred as the insider's game.
Is it the true or just a plausible explanation I do not know.

Anyway I prefer to use more marginal patterns for that reason.

The Law of the charts of Joe Ross is a very good choice.

The Wolfe waves is another good choice.

You can refer to the excellent book Trade Chart Patterns Like The Pros
by Suri Duddella. As he says you need only one pattern to be successful. In this book you can find reference for many patterns in a consistent way.

What happens in the market?

My theory is that in the Forex markets we have long term memory chaotic processes. And then it is possible to accept that what happens now determine the future.

So basically the price movement can be persistent (fractal dimension less than 1.5) or anti-persistent (fractal dimension bigger than 1,5) according to its fractal dimension.

On the market those phases can be seen easily.

-So when a fractal break - out starts we have a transition and during this transition patterns are formed that will indicate how far this break - out is going to go. I mean the Fibonacci relationships.

-When we are into a range-bound and anti-persistent movement often the Wolfe waves give us a clue where the price may go when it goes out of balance.

-Often after the fractal break out the movement starts to consolidate there often we see harmonic patterns to develop.

Forgive me folks I do not have pictures and shots for today. This is only abstract and theoretic stuff.

High frequency algorithms versus human traders

The topic of the high frequency algorithms is a very hot potato. In fact there is a phenomenon called the rise of the machines. The stock market nowadays is totally dominated by the high frequency algorithmic trading. The old school of the technical analysis is facing hard times.

What is the difference between the high frequency algorithms and the human traders.
In fact the main difference is that the human traders are not cooperative between themselves buts the high frequency algorithms can cooperate in a blink of an eye and create strange moves.

Yes the market may seem the same but an experiences eye can distinguished that its structure is different. It does not behaves the same way like before, the risk level and the volatility is much higher. The volume in the stock market is somewhat artificial. The Forex market is different of course.

Is it possible to use the algorithmic trading trading to the benefit to the medium retail trader?
Well I think that with the fractal dimension analysis we have some tool that we can use.
The exposed ideas have the task not to be a holy grail (not at all in fact) but to reestablish the balance that has bee perturbed by the algorithmic trading.


Risk 1: A systemic risk of market failure

The high frequency algorithms are a source of systemic risk of the market. Remember the flash crash of May 6th 2010? In fact those algorithms work like turbo accelerators of the movement.
The common approach is to look for a software bug, but I think that there is much more than that in the darkness.

My hypothesis is that a proper way to analyze the market is to analyze it as a multidimensional phase space of possible solutions. And what happens? It happens that sometimes this phase space is not too difficult, or in other terms is in the reach of the computing capabilities of the high frequency algorithms. What happens then? It happens that those machines combines their effort and start to cooperate together and all this in a blink of an eye. This is a hypothesis could be valid only when we have many competitive machines and not only one mega machine dominating the whole market.

Risk 2: Point of no return. Are we there?
We have to cope with this new reality. This is true because if they get out of the market the market will collapse in a blink of an eye.

Prediction versus Observation

A modern market approach consists in not trying to predict based on the gut feeling, technical analysis or even fundamental analysis.

The idea is to try to predict not the market direction but the market state.

1. Cluster the market states:

All the successful traders have in common that they try to distinguish the market state.
A common approach is to use the relationship between the time of the day in the forex markets and the corresponding volatility.

2. Appropriate Use of an adapted system for the period

So when a market state is identified it is necessary to use an adapted system for this market state.

Traditionally there are:

2.1 Intra-day Trend following system

A Moving average strategy is the archetype of those systems.

Neural nets perform very bad as a trend following systems (with some exceptions for Neural Nets specially tuned for those conditions).
Digital filters with genetic optimization do much better


2.2 Intra-day Range Bound system

Oscillator based strategy is the archetype of those systems. Modern oscillators appears to be much smoother but still all the oscillators shares a collinearity.

The neural nets perform well in those conditions.
Statistical indicators performs very well too. In fact the phase space is so big under such conditions that it is more appropriate to model it statistical instruments. The Bollinger bands as a rudimentary statistical instrument is a classic player in the Intra-day range bound systems.


2.3 Break - out system after a contraction of the volatility

Those systems are particular. For example a Neural Net cannot never ever predict a break-out in the forex market.

How to distinguish between those market conditions:

Well the obvious reason is practice, practice, practice and experience, experience, experience
The human mind and the human perception abilities are much more suited to distinguish those states than any machine. For example an image recognition, the machines are still not capable of doing it for now.

There are some methods and some new ones that will be covered later.

My opinion is that the human mind is appropriate to distinguish between the market states. We believe that when a market state is identified it will continue for a while.

Second we use statistical instruments and methods to help us.

And third once a state is identified we use the appropriate tested methodology. For me a tested algorithmic system appropriate for a particular market state outperforms the average human trader.

So we should not try to predict the future. Even if we try to predict this will destroy our capacity of observing. This phenomenon happens because we select and choose to observe only what conforms our hypothesis.

When the prediction is useful?

In fact the prediction is useful when you manage your position once the position is open.

Example 1: Trend prediction

You open a trade in a trending environment for example after a break - out. Your setup was good (a setup is a setup) and once your position is opened and the stop loss is returned to zero, your prediction game starts.

Now you are trying to predict the market, what is going on, what will happen in order to manage a position.

Remember, this is like chess game. Your little position can be like a spawn that can turn up to be a queen, when it reaches the end of the chess board. And in order to be a queen you have to push it forward. And as in the Chess a group of spawns have a better chance to reach their goal, to become a queen.

And vice versa as Glenn Neely observes even if you have a very good prediction capability this will not help you in your trading (even if you are right maybe your timing was wrong, or the stop was hit just before your prediction come true). You need a good setup.

Example 2:
You have a setup in a ranging environment. You buy at an oversold level. Logically you predict that it will reverse when it goes up at an overbought level. And this is prediction after you have opened a position.

Conclusions:
There are two types of trading styles. Those strategies have advantages and disadvantages.

One trading style is based on the prediction: Elliott wave analysis is the best example.

The second trading style is based on a behavior of the market with statistical tools: the price action setups are one of the best examples. For example a break - out strategy in a period with high volatility.

A third strategy is a mixed strategy between a behavioral and prediction. You use your setups as a behavioral strategy and try to manage your winning positions by a prediction strategy.

The advantage of the behavioral strategies is the good win to loose ratio.
The advantage of a prediction strategy is the good R/R ratio.

A mixed strategy tries to combine both.

New dimensions in the trend following philosophy

I personally think that the old school of technical analysis is unproductive in today Forex markets.

The only thing that remains valid is the trendiness. The markets looks similar but it isn't.

In order to use the power of the trend a mechanical and algorithmic system is necessary. It is necessary to have mechanical signals for a trend following strategy. And those signals need to be back tested.

This is not new and a lot of systems work on that basis. What is difficult is to be in the trend and to spot a trend as early as possible.

A recent idea is to use the fractal dimension as a market sentiment indicator. Classically it is used the ADX. But the ADX gives us just a measure of the actual state of the market. It does not see the future in the forex market or whatever market.

The fractal dimension indicators give us probabilities and how they change. This approach is new in the domain of the market following methodology. And we have to use it at its full extent.

We have an experimental system that tries to do that. You can download it from this blog:

http://microhedgefund.blogspot.com/

Fractal technical analysis

How to trend the extremely volatile forex markets. Do we have an edge in the forex markets?The fractal technical analysis can provide some help in this forex adventure. Here I show you some of my observations. We have two graphs, the first are drawn only trend lines. The second are plotted trend line but in combination with FGDI (I like to call it Fiji ;)) and the variation of the Hurst exponent. Do you see the difference?

Then I have plotted a rectangle with the area in which we ask do we have a trend or range? The Outcome of this is very important because from this it will depend what strategy we use.
And here on the second graph just shows us there is the FGDI with blue dimension. And when we have mainly blue dimension it is likely to have precisely Range for me there is a correlation greater than 0.5. However each Range not necessarily be exactly horizontal even I think this is an exception. Range quite often is positioned somewhere up or down and that makes frequent (almost certainly) its confusion with trend.

So what? We have a clear - and lower lows and lower highs?

Do we have the perfect trend lines that I have formed?

The difference is fundamental. When we have a Range we expect a break out, and when we have trend we expect a consolidation. Everything works, well almost, works in trend. In Range market that is not true.

Again, I repeat, we do not use the term trend in terms of technical standard anallysis. Maybe I will try to make some basis definitions of fractal technical analysis:

Strong Trend: directional movement of high volatility characterized by low fractal dimensio less than 1,5

Weak trend: directional movement of medium volatility characterized by low fractal dimension less than 1,5 (often an element of a wide Range from - a large order), sometimes making the same price patterns as broad Range pattern.

Wide Range: Top – often making sinusoidal movements with small frequency and average volatility characterized by a high fractal dimension over 1.5

Narrow Range: Most often sinusoidal movement with high frequency and low volatility characterized by higher fractal dimension above 1.5: Expect a break out.

It is very interesting because there is mixing concepts. For example, as wide Range and the weak trend can have directional movement and can be confused by the standard trend technical analysis (even a flat Range is an exception). But these the weak trend and the high volatility range are different things.

Which in this case I consider we have a high volatility range suitable for trading with Range strategies: Bolinger bands (the price rarely go beyond two standard deviations of Bolinger Bands makes it very safe entry points in case of range, I we do have a storng trend (high volatility with low fractal dimension) this is related to the fractal hypothesis: fractional Brownian motion http://e% 20n.wikipedia.org/wiki/Fractional_Brownian_motion maybe I do not express precisely but the coefficient of variation of such a distribution is infinite, this shows how and why you see the price going beyond the two standard deviations of the Bollinger bands.
If we have range we can use range tactics: oscillators, levels of support and resistance, Price Action, whatever you think will work in range.

By the second picture I will enumerate the patterns we have discovered:

Pattern 1
Breakthrough of a blue zone in red, it is often associated with transition from one phase to another from range to Trend. This is the most important pattern of the fractal analysis.

Pattern 2
Second Peter is associated with positive variation of Hurst exponent. This is a fine tuned analysis of the FGDI. And again if we see a peak we have to be waiting for something to happen.

Pattern 3
The Third pattern I have not drawn but is associated with expansion of channels of standard deviation of the FGDI. This means a strong movement

Pattern 4
Look at he Hurst difference. When we have a trend it is smooth, when we have a range we have a ruffled line of the Hurst difference.
Characteristics of noise

The last thing that I want to clarify is that when we have strong red dimension we can observe the phenomenon of black noise. This means an abrupt change of direction without any consolidation, technically Peter 1-2-3. No cups, no double, triple bottoms. Just a sharp price drop from one direction to another. In consolidation and Range have pink noise as the price turns mad and eat your short stops just before to continue in the right direction.

Let's look at the graph:
Naturally, in the graph the trend movement seems to have started i arround 15:00 on 2 of June. And I ask myself, was I blind, that I did not see it. But that was so easy, why not I did not see it.
But fractal analysis shows that there is a trend component of the range. The Real trend starts June 2 at night and at the beginning of June 3 and its scale is very limited (and anyway if we plot afterwards the trend lines it looks that it has begun much earlier,) (Moreover the fractal dimension varies very long in the afternoon around 1.5, which means that the movements can not be predicted, everything is possible),
The beginning of the trend movement is made by a fractal break-out from blue zone to red zone. This happened on 3 June in the morning. We see clearly the fractal breakout. And in the same time we have a peak in the variation of the Hurst exponent.
After that the dimension increases and we enter a period of adjustment and this is the Range of the morning of 3 June.
Technically the correction can be analyses as a Ross Hook, with the possibility of the TTE (Traders Trick Entry). And indeed it turned out that the break out is false.
From a perspective of fractal analysis we see it in real time. We observe an increasing dimension and reducing the variance of Hearst. The up movement was to go nowhere

For this kind of phenomena the standard technical analysis has no explanation. The trend lines are often misleading. But now we have the fractal technical analysis LOL.
If these assumptions are correct we have a new and very powerful tool for technical analysis. These ideas are super fresh, and now we must verify them. The positive point is that the fractal approach provides an element of objectivity, which may improve the technical analysis. And it gives an edge – clarity in the real time technical analysis. Afterwards everyone can make a precise technical analysis.

The idea behind this blog

Do we all search for a holy grail in the Forex market? Do we all search for an edge?The answer is definitely Yes !!!

This blog was born because of my desire to free myself from the ideas and toughs in my mind in a systematic way. The only way to get out was to write them down and make them public. As this blog will be not an ammunition for a spamming attack if you find it it will be not a chance but in purpose. I hope you will make a good use of the ideas and have fun. There are another three blogs. One blog is an experiment for trend following micro hedge fund (with 100 USD a good joke isn't it) and the other concerns strategies for day trading the forex market. Those three blogs form a whole, but this blog is more general and discuss general ideas and theories.

There is an ongoing continuous research, some of the brightest human researchers are working on that. There are great traders, programmers, mathematicians, economists they all work on the same topic.

It is not possible for a simple human perspective to be in the same time, mathematician, bright program coder, a great speculative trader with innovative ideas. We all can have some domain of competence but to really have an edge we need to be all that. Is it possible? Are we super humans heroes? Of course it isn't possible but we can use the knowledge and the brainpower of all them to have a hack of the market. That hack cannot beat all the times the market but we could exploit opportunities if we have a back door on the market.
During my research I found on the net a lot of interesting materials that I really want to share with my friends. That is why I created this blog.

1. The idea is that we have to combine many, many different approaches in order to have an edge. If we use only one trading strategy, my opinion is that we cannot have an edge in the Forex market.

2. The second idea is that we need as much automatic analysis as possible. We want our brain to make a synthesis and not an analysis. This idea includes the idea that we need a reference trading system.

The system I have chosen for me at the beginning was the Bill Williams trading system. I really like it and I made some personal improvements. But still not profitable. After that i have read the e-book FXWizard by Rob Walton. This system was easy to understand, logic, beautiful but unprofitable.

Nevertheless it was not profitable at all in day trading.

After that I have visited every forum you can imagine but ... you know the story.

So after much research, this sounds familiar, I had some ideas that I wish to share in this blog.

I decided to get out of the conventional wisdom and common sense and to explore new territories. As Dr. Bishop in Fringe says: Accept the impossible and maybe you could touch the truth.

I challenge the conventional truths like

-the technical analyse patterns use the psychological moods: a wrong concept

in fact the patterns can have different origin. My theory is that it is wrong to use the technical analysis theory developed from the stock market and apply it automatically in the Forex.

The big figures on 4 h times frame are mostly due to market sentiment, but they can be a pure chance. But once they are formed everybody looks at them and it becomes really freaking risky to day trade them with a lot of leverage and big lots.

There are other figures that are due to the change of the volatility based on the daily session.

In that sense a pattern is cause by the volatility. And the daily volatility has cycles that can be observed, statistically quantified.

The figures on the small times frames can have different origins. They can be random or provoked by some cycle chaotic attractors or again created by intra-day volatility collapse .

-the usefulness of the Elliott wave principle as a wrong concept in the modern Forex markets

etc.

The Elliott wave principle has its source in a simple game theory concept for the balance of power between bears and bulls, provoked by the balance between the human emotions of greed and fear. Dear readers today in a market dominated by the algorithmic trading and high frequency algorithms don't you think that the premises are missing. THEY ARE NOT THERE.

The model was interesting it was based also on statistical observation, that the rise continues longer than the fall. But now and today it does not holds as a predictive instrument.

My opinion was that if it is useless ot can do no harm as you use it to find an entry solution in the trend. But please don't be eaten alive and do not use it to make a counter trend prediction.

But the gurus, as Prechter or Neely find themselves very smart and use it to predict the market reversals. Well when you make such a prediction sometimes you will be correct. And then the crowd sees only the last "prediction", but those before are easily forgotten. But Neely saw that there was something missing and recognized that his predictions are not correct more than 50 % of the time. So he uses a secret method calls a River theory, that has nothing to do with any wave principle.

The main ides in this blog us the use of market observation of the fractal dimension of the price times series as a leading concept.