Statsmodels and Prophet
1 Like
Itâs not AI but I think this is the type of pattern recognition I think youâre looking for: GitHub - neurotrader888/TechnicalAnalysisAutomation
Time series forecasting and black sholes model⌠read about them⌠might be helpful.
Seems to me, predicting anything off a price chart is like predicting movement of a shadow when both the object (i.e underlying instrument idiosyncratic factors) and the lighting (i.e. macroeconomic factors) are moving around. If either the lighting or object is stationary, AND you are observing the non-stationary item, then you may get accurate in predicting direction or change in direction (i.e trend) itself.
Thanks
Thanks
There are already trend changes indicators (not invented by me) that I want to identify using AI. I have been already doing this manually for years (by looking at the SMA in the charts), the approach is doing the same but instead of manually with AI.
https://otexts.com/fpp3/ uses r, but is probably more than you want to know about time series. some of the arima stuff is available in python.
1 Like
To get started with your AI-powered stock market trend change identification project, you can follow these steps:
- Collect and Prepare Data
Market Data: Collect historical stock market data. This data should include open price, close price, highest price, lowest price, and trading volume. You can use APIs like Alpha Vantage, Yahoo Finance, or other financial data sources.
Additional Data: If needed, you can add additional technical indicators (RSI, MACD, Bollinger Bands, etc.) or economic factors.
- Data Processing
Data Cleaning: Remove or fill in missing values, check data consistency.
Data Normalization: Normalize or min-max normalize values ââto ensure all features are on the same scale, which is important for many machine learning algorithms.
3. Choose a Model
Temporal Model:
LSTM (Long Short-Term Memory): Great for time series data as it is capable of learning long-term patterns.
GRU (Gated Recurrent Unit): Similar to LSTM but with a simpler structure.
Classification or Regression Model:
If you want to classify trends (up/down/sideways), you can use models like Random Forest, SVM, or Neural Networks.
4. Libraries and Tools
Python: Is the main programming language for this project as there are many supporting libraries.
Pandas: For data processing.
NumPy: Arithmetic calculations.
Scikit-learn: Basic machine learning tools.
TensorFlow or PyTorch: For building and training deep learning models like LSTM.
Keras: High-level interface to TensorFlow, easy to use for building models.
TA-Lib: Technical analysis library for calculating indicators.
5. Model Development and Training
Data Splitting: Splitting the data into a training set and a testing set.
Model Building: Use the selected libraries to build your model. For LSTM, you will need to reshape the data into 3D [samples, time steps, features].
Training: Train the model with the training data and tune the hyperparameters.
6. Evaluation and Backtesting
Evaluation: Use metrics such as accuracy, F1-score, or mean squared error depending on your problem (classification or regression).
Backtesting: Use historical data to test the performance of the model. This helps you understand how your model might perform in real market conditions.
7. Fine-tune and Optimize
Based on your testing results, you may need to fine-tune your model, adjust your indicators, or even change the model entirely if the results are not as expected.
8. Deploy
Once you are satisfied with your model, you can deploy it to analyze real-time data or in simulations.
Note:
Overfitting: Be careful about overfitting your training data. Use techniques like cross-validation to ensure your model generalizes well.
The market is not perfectly predictable: Your goal is to learn and test, not to predict the market accurately.
Good luck with your project!
1 Like
Indeed.
Thanks
I recently did something similiar for my Final Year Project at the Uni. I used LSTM to predict the prices and did some feature engineering as well like adding MA and EMA. I would recommend doing some research and adding alot of different features( technical analysis and economics would be your best bet) and then use particle swarm to choose the best features for prediction.
The reason for this suggestion is in my model I had a massive feature set and my model was overfitting alot, especially when the stock didnât have a particularly long history.
Feel free to reach out if you need some help if you decide to use this method. Youtube should have alot of guides to help you on the way.
@HemsankhaRoy Particle swarm ? What on Earth is particle swarm ?
Particle Swarm optimization
reading Interesting, never heard of it.
Thanks
@HemsankhaRoy @Hector_Gonzalez
I see this topic is garnering a lot of interest. Machine learning tools perform best on data that is statistically static. Meaning the inherent characteristics and structure of the data doesnât change from period to period. This is what gives ML algorithms their predictive powers. Market data is very noisy.
Which explains why the time frames that the quantitative funds are involved in are in milliseconds or, seconds at most. Which we call high frequency / market making funds. They just make markets for everyone to buy and sell, and earn a commission on the fills. The forces driving the price action from 1 second to another are very stable. The environment doesnât change.
When you are dealing with longer term data, daily price action for example, the market environment could be different based on new economic data, companies reports, geopolitical news, market dynamics (end of month, end of quarter, one hedge fund is liquidating for unrelated reasons, âŚ), among many others.The interplay of all of these different forces at different times can create different price action. The data is very noisy and prone to overfitting.
Just my opinion. I think that the first task should be to make sure that the training data is close to testing data, and to future expected data. Otherwise overfitting is very common.
2 Likes
Yes, and as you mention, at very small time intervalsâŚarbitrage windows as short as 10 seconds. Michael Kearns from UPenn has been studying this domain for a long time. Anyone planning to enter this space should read his papers to understand what theyâre up against. For exampleâŚ
Machine Learning for Market Microstructure and High Frequency Trading
1 Like
@ai_curious
Great paper, thank you. But I wouldnât encourage any retail trader to try this path. Hedge funds with tens of millions of dollars canât compete with Citadel and Renaissance because these guys have billions of dollars to deploy in billions of trades, closer servers means their orders get there first, and hiring an armada of PhDs get them very robust and diversified algos.
There are other ways to get involved that leverage individual strengths. Arms race as a quant isnât one, unless one wants to work for them. 
Edit :
I hate to discourage anyone to try anything. Just wanted to shine the light on this topic.
2 Likes
Couldnât agree more. I linked that as a cautionary âdonât try this at home kidsâ not as a DIY cookbook or to contradict your reply above. Guys like him have been building models like that for resource elite orgs like Goldman, BofA, et al for decades. Itâs not a fair fight.
2 Likes