Ref. cost intuition, the direct costs linked to the use of a GEN-AI service (ex. ChatGPT 3,5) were used. But should we also account for the computing and other costs required in order to build a Gen-AIN model? What other cost should one reasonably add in order to estimate th all-in costs?
1 Like
Yeah. Prof Andrew was looking at the monetary cost for end users.
If you expand the definition of the cost in this way, we may even look at opportunity costs too in terms of letting go of actual people who were performing the tasks prior to the model. Also, the cost of training the model, the impact of the energy used on the environment (maybe the carbon footprint is large and made our world hotter). The list is definitely super long.
2 Likes
Gen AI projects, beyond using a ready made solution, are complex software engineering projects. There are several cost categories to be considered:
Development Cost:
- User Requirements/Product Management
- Software Develpment
- Testing & Quality Assurance
- Development Infrastructure
- Project Management (~15-20% of total)
Operations Cost:
- IT Infrastructure (compute, storage)
- Operations personnel
- User Guide & Support staff
- Third Party modules licenses/service fees (e.g. OpenAI LLM API, *)
- Quality monitoring & assurance
To reduce overall cost & delivery time for AI projects it is advisable to leverage ready made AI platforms, so called MLOps or LLMOps. These provide a baseline infrastructure for development & operations that offer common functionality out of the box (e.g. to facilitate common tasks like model training, validation and monitoring)
*) this is the cost factor that Andrew was pointing out
1 Like