Making Generative AI Work for Your Enterprise – Excerpts from Recent LinkedIn Live Event

• Artificial Intelligence

Generative AI and Large Language Models (LLMs) like GPT have the potential to radically transform the way businesses use data and analytics to operate, innovate, serve customers, and grow. LLMs and existing enterprise systems and processes can expedite access to impactful insights and enable personalized content generation for enterprise users and their customers.

The promise of generative AI (Gen AI) is vast. But leveraging the technology in the day-to-day business of the enterprise and driving significant improvements to key business parameters requires a clear definition of what we can expect from the technology and the metrics we plan to use to measure the success of a Gen AI initiative.

I recently hosted Course5’s LinkedIn Live on ‘Unlocking the Power of Generative AI for Your Organization – A Q&A Round Table on Large Language Models (LLMs)’, an event from the Course5 Compass – GenAI Series, with my co-panelist – Nitesh Jain and Jayachandran K R who provided practical considerations for assessing the actual value of enterprise-level investments in new AI technologies like Gen AI.

Here are some key takeaways from the roundtable event –

How do you measure the success of a Generative AI/LLM deployment?

Defining the right KPIs is the first milestone in measuring the success of a Gen AI implementation.

At a strategic level – Rather than looking at direct ROI, it would be more beneficial to use metrics that measure the impact of the technology on other data & analytics investments and the business. Four key performance indicators (KPIs) you could look at are –

• Adoption
  Organizations are already generating much data and insights, but not enough people use it. Is the LLM driving higher adoption of data and insights in your organization?
• Deployment of human resources in more meaningful and impactful activity
  If Gen AI can reduce time to data and insights, can people focus on higher-level tasks that add value to the business?
• Experience enhancement
  Does it improve the experience of accessing a service for customers and employees and the outcomes of that service?
• Ability to serve customers faster and better
  Does the LLM enable quicker and more relevant information retrieval and help serve customers better?

At a technical level – If you have to measure the success of a Gen AI initiative, you need to consider whether you are looking for a supercharged performance or an incremental improvement to your existing models/systems already in business-as-usual mode.

If, for instance, you’re looking at replacing your existing chatbot with one powered by LLMs, you would look at a very granular set of metrics as – Is this improving the latency, response time, accuracy, user experience, etc. of my chat service? Is it creating a very engaging and exciting experience for my customers?

Enterprises already have several data and analytics investments. And many Gen AI tools and platforms are coming up every day. What is the right approach for adopting LLMs?

Organizations that have adopted technology without figuring out how it fits into their existing ecosystem have failed to deploy or embrace it. A tremendous amount of investment has gone into the current systems, already driving some business value.

Also, when you think of leveraging LLMs within this system, you can use models from providers like Azure, Google, and AWS; apps, APIs/plugins, and Chrome extensions; or enterprise platforms and open-source finetuned models.

To choose the right approach, you must be clear about what you expect from the technology. Augmentation is the fundamental principle to consider while defining Gen AI/LLM technology expectations.

Augmentation using LLMs can happen at different levels –

• Basic
  Augmentation can turn dry data and standalone insights into narratives that connect the dots in the business.
• Intermediate
  Augmentation can drive interaction and engagement with the data, so a user can drill down and ask for more refined information to make/test a hypothesis, find causal connections, derive predictive insights based on actual/simulated data, etc.
• Advanced
  Augmentation can provide business and R&D teams with new ideas, product suggestions, and questions they haven’t considered! Here the AI program acts as a research assistant, suggesting possible scenarios and things the user should care about during strategic planning.

Using Baselines & Benchmarks

Your current business performance can act as a baseline to judge performance improvement or augmentation, and every model will have its performance benchmarks. Consider which approach will bring you substantial incremental gains on your current baseline and has the scope to scale. Then pick the right path for you.

What you can get from an out-of-box model from Azure, GCP, or AWS versus an enterprise model will be very different, for instance. You can start with the open source models, then assess where you need prompt engineering, finetuned models, larger systems within which Gen AI is integrated, and so on.

Can you give us some use-case examples?

Generative AI is such a powerful technology that teams across the enterprise may be looking to leverage it in some way. They could search massive data sets in seconds, intelligent summarization, content generation, emotional analysis, sentiment analysis, topic modeling, etc.

Data Science teams can use Gen AI copilots.

R&D and Insights teams sitting on a wealth of structured and unstructured information can use it to provide topics and supporting data required for any specific R&D initiative.

Marketing and E-commerce teams can use Gen AI to generate personalized, customized product content and messaging. By combining brand messaging guidelines with product data, you can automate intelligent custom content generation with inbuilt SEO optimization for better customer traction.

The Pharma industry can leverage Gen AI to cull critical insights from new information on adverse events – in the context of the vast repertoire of existing knowledge on molecules and disease/therapy areas.

The scope of use cases is relatively limitless. To prioritize your use case, you must assess the feasibility, likelihood of adoption, and the extent of impact expected in each situation.

The whole point is to make more ingenious tasks easy for your employees.

What kind of Audit process is required to ensure my benchmarks stay true or improve?

Model audits can be done at various levels – at the core model (model performance validation) level, the model’s application context, and the overall governance of all LLMs in an organization.

Auditing can also happen at the level of the data, that is, what kind of data the model is trained on. This is important because, eventually, a model’s capability is only as good as the data it is trained on.

Most importantly, it would be best if you constantly validated the output of a model with the ground truth. You’ll need to assess the work carefully, give it scores to evaluate its quality and accuracy, and determine if that level or percentage is good enough for the task.

Moreover, these models are constantly evolving. You will need high rigor through processes like model versioning, prompt versioning, etc., to closely monitor and refine their performance over time.

The Larger Picture

Community backing is one of the most critical factors in successfully selecting and deploying any new technology. The community creates excitement about technology, makes iterative improvements, and propels an organization’s innovation agenda forward.

At Course5 Intelligence, we aim never to provide analytics and AI solutions. We endeavor to understand the community that uses our platforms and help enterprises drive adoption within that community, which leads to business impact. For instance, our Augmented Analytics solution using Course5 Discovery, integrated with GPT models, involves a fundamental User Journey Analysis – who consumes the insights, in what form, how frequently, to what purpose, and so on. And then, we use LLMs to make this entire user journey more intuitive, leading to the desired business outcomes. Course5 Compete, a market, and competitive intelligence platform, uses LLMs for personalized product content generation, SEO, product content data reconciliation across diverse channels, and many more.

Generative AI becomes the means to a larger goal.