Why all those complex Artificial intelligence models fail predicting the markets?

The point as I see it is that predicting the markets using any AI brute force (force method in the sense of relying on the abilities of the algorithm by itself) is a dicey business.

The markets are terribly complex they constantly are looking for balance and falling out of balance. The markets are so terribly complex that you may wander if the complex learning paradigms can beat the buy and hold on one hand and the simple linear regression on the other. Most of the time they can't pass the Monte Carlo simulations.

If that was not the case every AI specialist would find easily his way to riches. The problem is not in the AI methods they work fine on normal problems. The slight nuance is that you are asking them to do impossible things. As they can't let you without an answer they will give you an answer to the question as it is in the book Hitchiker guide to the galaxy. They asked a very complex computer what is the purpous of life and the answer was 42.

So the direct and brute force approach of successful and sound and proven AI do not work consistently enough.

So what can be done. From one hand those guys are hired by the HF quantshops where an edge is found in the speed of execution. Then the edge needs deep infrastructure understanding and market structure understanding that are completely different fields of knowledge. Only by combining this meta knowledge those quant shops succeed.

Other possible way is to look towards econometrics and deep market knowledge. Then you may be able to find a good definition of the problem and apply correctly the model. The economist need to define the problem the AI specialist has to apply correctly the model.

Those are two main models I know that lead to an edge. There may be other ways in order to beat the markets but they are fringe like entropy analysis, market states clustering and chaos theory etc.

So then why people do stay?

One possible answer is that beyond profitability the quest is very interesting per se, you learn new things, it is an excellent "game" for smart people. Eventually you find good on-line friends to work together on projects thinking that the grail is right next on the corner. Everything will be fine unless you allow yourself to make a gross loss.

Start small start smart. Take it easy.