AI pilots and proofs of concept are everywhere these days, but getting those projects into real-world production is where many teams seem to hit a wall.
I’ve seen organizations struggle with everything from messy data and system integration to security requirements and getting teams to actually adopt new AI tools.
For those who have worked on AI projects, what has been the biggest challenge in your experience?
Was it more of a technical issue, or did business processes, governance, and organizational buy-in end up being the bigger hurdle?
Curious to hear what others have seen in practice.
In my experience security and scalability were the biggest challenge. AI can build tons of new functions and they might actually work, but when you contrast with the industry standards you realize the flaws that it has, on the same page when you try to add a new feature a bunch of other features that previously worked start to malfunction. From my perspective it was more of a technical issue that need to comply with business processes before deploying.
True @hynpbus, in traditional Machine Learning, you have clear metrics (F1 score, ROC-AUC). In Generative AI, defining and automating “correctness” or “safety” at scale without relying on incredibly expensive human-in-the-loop validation is a massive bottleneck
In my experience, apart from security and scalability, in practice there are other structural challenges too:
- GenAI projects are still finding where they stand in the ROI horizon. Many projects still struggle to justify their high development costs against actual value delivered
- A standardized framework automation to tune the hyper-parameters inherent in production RAG systems (e.g., chunk size, metadata filtering, and vector database retrieval optimization) - Curious to know what others are doing in this space.
- Measuring, benchmarking and guaranteeing deterministic, accurate outputs, especially when balancing an LLM’s role as an autonomous ‘tool user’ versus a core predictive engine.
Great points @parulgaba. In our experience, the biggest challenge is usually everything around the model rather than the model itself. Pilots run on clean sample data, but production data is scattered, restricted, and constantly changing. Teams also rarely agree upfront on what “good” looks like, so a small golden set of real business questions, checked on every change, helps a lot.
Ownership is another gap, since monitoring, drift, and cost tracking often fall between data, IT, and business teams, and security or compliance reviews that come in right before go-live can delay projects by months. Finally, ROI gets tied to model accuracy when the business really cares about hours saved or faster turnaround.
For RAG tuning, we’ve had good results treating chunk size, top-k, and reranking as experiments tracked in an eval harness like Ragas or DeepEval.