Why Enterprise AI is Hard to Scale

Johnny Chan
7 min readNov 25, 2024

--

The short answer: no company wants to share data to build a unified model that benefits their competitors.

As an AI consultant working in the oil and gas industry in North America, I’ve observed the AI transformation firsthand. A few years ago, when AI began to gain momentum, there was a widespread belief that the industry was on the brink of a revolution. Companies across the sector started hiring professionals to develop AI tools and applications. While there have been some notable successes, many of these initiatives have failed to deliver on their promises due to various reasons.

Diverse Operational Approaches

Every organization operates differently, with unique processes, equipment, and workflows. A method or model that proves effective in one company may be entirely unsuitable for another. This variability makes it challenging to scale AI solutions across the industry. What works in one setting might not translate well in another due to these inherent differences.

Anomaly detection modeling was quite popular a few years ago around oil and gas companies for asset health management. Some of the projects require only basic statistical approaches, while others demand sophisticated deep learning models. Let me illustrate this with a real-world example:

Company A operates modern facilities with high-frequency sensors collecting data every second, making it ideal for deep learning-based anomaly detection. Their equipment generates rich, multivariate data streams that can capture complex patterns and interactions. In contrast, Company B operates older facilities where data is collected manually every four hours. For Company B, simpler statistical methods like moving averages or control charts might be more appropriate and equally effective.

This disparity extends beyond just data collection:

  • Equipment Configurations: Different manufacturers, models, and customizations mean that “normal” operating conditions vary widely
  • Maintenance Schedules: Various preventive maintenance approaches affect how equipment degradation patterns manifest
  • Operating Conditions: Geographic locations, climate differences, and production demands create unique operating environments
  • Risk Tolerance: Different companies have varying thresholds for what constitutes an “anomaly” based on their risk management strategies

These differences mean that a one-size-fits-all AI solution is rarely feasible. Even seemingly similar use cases, like pump failure prediction, require significant customization for each implementation. This customization isn’t just about tweaking model parameters; it often involves fundamental changes to the approach, data preprocessing, feature engineering, and alert thresholds.

The challenge is further complicated by the fact that many vendors and solution providers market their AI products as universal solutions, when in reality, successful implementation requires extensive adaptation to each company’s specific operational context. This mismatch between expectation and reality often leads to disappointment and skepticism about AI’s value in industrial applications.

Lack of Subject Matter Expertise

Models cannot be developed solely by Data Scientists, but require significant insights from SMEs who understand the underlying physics and operational complexities. In my experience, securing SME involvement was consistently the most challenging aspect of any project. Even when working directly with client companies, they often proved reluctant to allocate SME time for critical tasks like data labeling or troubleshooting, primarily due to the substantial effort involved.

This challenge manifests in several ways:

  1. Time Allocation Conflicts:
  • SMEs are typically senior personnel with critical operational responsibilities
  • Their primary duties often take precedence over AI projects
  • The opportunity cost of pulling them away from operations is substantial

2. Knowledge Transfer Bottlenecks:

  • Complex industrial processes require years of experience to fully understand
  • Critical insights often exist only in SMEs’ heads, not in documentation
  • Translating domain expertise into features and rules for AI models is time-intensive

3. Continuous Engagement Requirements:

  • SME involvement isn’t a one-time effort but a continuous necessity
  • Model maintenance and updates require ongoing SME validation
  • New edge cases and operational changes demand expert interpretation

A typical scenario I encountered involved a major oil company’s attempt to develop a predictive maintenance model for their compressors. While the data science team could build sophisticated models, they struggled to identify which patterns were genuinely concerning versus normal operational variations. Only the SMEs could provide this crucial context, but they were already overwhelmed with their daily responsibilities.

Best Practices I’ve Developed:

  1. Structured Knowledge Capture:
  • Schedule regular but brief SME sessions
  • Document all insights systematically
  • Create knowledge bases for future reference

2. Efficient SME Time Usage:

  • Prepare specific questions in advance
  • Use batch reviews instead of ad-hoc requests
  • Leverage recorded sessions for training new team members

3. Hybrid Team Structure:

  • Pair junior engineers with SMEs for knowledge transfer
  • Create liaison roles between technical and operational teams
  • Develop internal training programs to grow expertise

Many companies ultimately abandon their AI initiatives due to the inability to secure or maintain adequate SME involvement. This challenge isn’t just about having the budget to hire qualified experts; it’s about creating sustainable processes for knowledge transfer and ongoing support. Without this foundation, even the most sophisticated AI models risk becoming irrelevant or, worse, dangerous if they fail to capture critical domain knowledge.

The Challenge of Data Sharing

The most significant hurdle, however, is the reluctance of companies to share data. In theory, many of the challenges mentioned above could be mitigated if companies were willing to pool their data and collaborate on building unified models. But in practice, this is unlikely to happen. In the oil and gas industry, data is considered a valuable trade secret, representing a competitive advantage that companies are unwilling to share. No organization wants to help a competitor by contributing to a model that could benefit the entire industry.

This resistance to data sharing manifests in several critical ways:

  1. Competitive Intelligence Concerns:
  • Equipment performance metrics might expose technological advantages
  • Maintenance records could indicate cost structures and operational strategies
  • Well data and reservoir characteristics are considered crown jewels of information

2. Legal and Contractual Barriers:

  • Complex data ownership agreements with service providers
  • Intellectual property concerns around proprietary processes
  • Regulatory compliance issues across different jurisdictions

3. Failed Collaboration Attempts:

I’ve witnessed several attempts at industry collaborations that illustrate these challenges. For example, a major initiative to create a shared database for equipment failure predictions fell apart when companies realized that sharing such data might reveal:

  • Their maintenance practices
  • Equipment reliability metrics
  • Operational efficiency levels
  • Cost structures and operational weaknesses

4. Impact on Innovation:

The reluctance to share data creates a paradoxical situation that

  • Each company must solve the same problems independently
  • Resources are wasted reinventing solutions
  • Overall industry progress is slower than necessary
  • Smaller players struggle to develop competitive AI solutions

5. Alternative Approaches That Have Emerged:

Some companies have tried to address this through:

  • Anonymized data sharing consortiums
  • Third-party data aggregators
  • Synthetic data generation
  • Federated learning approaches

However, these alternatives often fall short because:

  • Anonymization can remove crucial context
  • Aggregated data loses company-specific nuances
  • Synthetic data may not capture real-world complexity
  • Federated learning still requires significant coordination and trust

The result is a fragmented landscape where each company builds its own isolated AI solutions, often with suboptimal results due to limited data. This situation is particularly ironic given that many of the technical challenges in oil and gas are universal — from equipment maintenance to production optimization. A collaborative approach could accelerate solutions to these common problems, but competitive pressures make this virtually impossible.

Enterprise AI Applications Need to be “Right-Sized”

Customers often approach consultants with ambitious requests to build comprehensive AI applications that solve all operational challenges through a single dashboard. This was — and still is — a common misconception about AI’s capabilities. However, my experience has shown that successful enterprise AI implementations typically start small and focused.

Here’s why the “right-sized” approach works better:

  1. Manageable Scope: Starting with specific, well-defined problems allows teams to demonstrate value quickly while maintaining realistic expectations.
  2. Faster Implementation: Smaller projects can be completed and deployed more rapidly, providing immediate business value and learning opportunities.
  3. Reduced Risk: By focusing on targeted solutions, companies can minimize investment risks and validate their AI approach before scaling.
  4. Better Resource Allocation: Limited SME time can be used more effectively when focused on specific use cases rather than spread thin across multiple initiatives.

Best Practices for Enterprise AI Implementation:

1. Start with High-Impact, Low-Complexity Projects:

  • Focus on use cases with clear ROI
  • Choose problems where data is readily available
  • Select applications where SME input can be clearly defined

2. Build Modular Solutions:

  • Design systems that can be expanded incrementally
  • Create reusable components when possible
  • Maintain flexibility for future integration

3. Establish Clear Success Metrics:

  • Define specific KPIs for each AI implementation
  • Set realistic timelines for value realization
  • Create feedback loops for continuous improvement

4. Invest in Data Infrastructure:

  • Prioritize data quality over quantity
  • Develop robust data pipelines
  • Implement proper data governance frameworks

Conclusions

while AI holds tremendous potential, scaling it across an industry like oil and gas is fraught with challenges. The lack of data sharing, coupled with the diversity of operations and the technical limitations of data infrastructure, makes it difficult to achieve the kind of transformative impact that many envisioned. The road to enterprise AI at scale will require not just technological innovation, but also a fundamental shift in how companies approach data and collaboration.

The key is not to aim for a full-scale AI solution but to identify areas where AI can provide sustainable competitive advantages within the constraints of limited data sharing and company-specific operations. This approach may not be as glamorous as industry-wide transformation, but it’s more likely to deliver real business value in the enterprise context.

--

--

Johnny Chan
Johnny Chan

Written by Johnny Chan

Co-founder of Hazl AI -- a platform for your one-stop AI and cloud services. Visit us at hazl.ca

No responses yet