I am building a totally local AI model. I started with OpenAI’s GPT -2 small from GitHub. Then shifted it into a larger model.
| Component | model |
|---|---|
| Architecture | GPT2LMHeadModel |
| Transformer blocks | 24 |
Hidden width / n_embd |
1024 |
| Attention heads | 16 |
| Head dimension | 64 |
| MLP/intermediate width | 4096 (GPT-2 default because n_inner=null) |
| Context | 1024 tokens |
| Vocabulary | 120,446 |
| BOS | 50256 |
| EOS | 50256 |
| PAD | 50256 |
| Embedding/LM head | tied |
| Activation | gelu_new |
| Dropout | 0.1 |
| Config dtype | float32 |
For a total of 426,696,704 parameters.
My data is clean; I manually cleaned it, used a different Cloud AI to check it, and I created a dedicated definition database. And I have trained on many different datasets from Hugging Face and my own creation.
It’s been some time, and I cannot get the model to cross the conversational boundary into human-like speech. I don’t know what I am doing wrong, if anything. The model has spent months on definition training, logical training, context, and causal training. I just started training with short stories.
Yet, honestly, I don’t know what I am missing. Any tips would be helpful.