AI pilots and proofs of concept are everywhere these days, but getting those projects into real-world production is where many teams seem to hit a wall.
I’ve seen organizations struggle with everything from messy data and system integration to security requirements and getting teams to actually adopt new AI tools.
For those who have worked on AI projects, what has been the biggest challenge in your experience?
Was it more of a technical issue, or did business processes, governance, and organizational buy-in end up being the bigger hurdle?
Curious to hear what others have seen in practice.
In my experience security and scalability were the biggest challenge. AI can build tons of new functions and they might actually work, but when you contrast with the industry standards you realize the flaws that it has, on the same page when you try to add a new feature a bunch of other features that previously worked start to malfunction. From my perspective it was more of a technical issue that need to comply with business processes before deploying.
True @hynpbus, in traditional Machine Learning, you have clear metrics (F1 score, ROC-AUC). In Generative AI, defining and automating “correctness” or “safety” at scale without relying on incredibly expensive human-in-the-loop validation is a massive bottleneck
In my experience, apart from security and scalability, in practice there are other structural challenges too:
- GenAI projects are still finding where they stand in the ROI horizon. Many projects still struggle to justify their high development costs against actual value delivered
- A standardized framework automation to tune the hyper-parameters inherent in production RAG systems (e.g., chunk size, metadata filtering, and vector database retrieval optimization) - Curious to know what others are doing in this space.
- Measuring, benchmarking and guaranteeing deterministic, accurate outputs, especially when balancing an LLM’s role as an autonomous ‘tool user’ versus a core predictive engine.