OCI Generative AI Integration with LangChain UseCases

Introduction

Oracle Cloud Infrastructure (OCI) Generative AI is a fully managed service that seamlessly integrates language models into various use cases, such as writing assistance, summarization, and chat functionalities. OCI Gen AI currently supports Cohere pre-trained models (command and embedding) as well as the llama-2 70b parameter model. My colleague, Rekha, has authored a blog post explaining the capabilities of OCI Generative AI service.

LangChain is an open-source framework designed for creating applications leveraging large language models (LLMs). Having been in existence for slightly over a year, LangChain has become popular framework for building LLM based applications. Recently, they released the stable framework version v0.1.0. This playlist comprises a series of videos highlighting key focus areas of LangChain v0.1.0 framework.

The OCI Generative AI team has collaborated with LangChain to offer OCI Gen AI LLMs within the LangChain framework. This integration allows for the effortless creation of LLM based applications. Additionally, LangChain offers integration with LLM models deployed on the OCI Data Science platform. For further details, please refer to this link.

In this blog, we will go through fusion application related use cases considering ecommerce customer has implemented fusion and now using OCI Generative AI with LangChain to enhance the experience.

Use Cases

1. Item Description Generation using AI

This use case illustrates the generation of product descriptions, key features, and benefits for a specified product name and its attributes by leveraging the OCI Generative AI large language model through the LangChain framework. This use case has already been incorporated into Fusion Product Lifecycle Management (PLM) in the 24A release, available under controlled availability.

The following code initializes the OCI Gen AI LLM from the LangChain library, providing model, endpoint, compartment, and model parameters. It then sets the prompt and performs inference to generate the product description, key features, and benefits for given product.


from langchain_community.llms import OCIGenAI

llm = OCIGenAI(
    model_id="cohere.command-light",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="COMPARTMENT_OCID",
    model_kwargs={"temperature": 0.7, "top_p": 0.75, "max_tokens": 1000}
)

prompt = """Generate a product description, key features, and benefits for a specified product based on its attributes.
Item: Adult Badminton Racket BR 580 Lite Blue Grey
Color: Lite Blue Grey
Balance Point: Head heavy
Weight: 78gm
String Tension: 20-24 IBS
Shaft Stiffness: Flexible"""

response = llm.invoke(prompt)
print(response)

2. Item Classification using Retrieval Augmented Generation

This use case demonstrates the classification of items into predefined categories using retrieval-augmented generation. The categories are transformed into embeddings and stored in a vector store. Similarly, the item query is converted into embeddings and categorized into the category through similarity search. LangChain integrates with the OCIGenAIEmbeddings model for embedding creation, which is based on the Cohere embedding model. In this scenario, the embeddings are stored in the ChromaDB vector store. Oracle has also introduced the Integrated Vector Database, you can read more about it by clicking on this link. This use case can also be adapted for item search or recommendations of similar items.

Oracle has recently introduced the OCI Generative AI Agents service in beta availability. This service combines the capabilities of large language models (LLMs) and retrieval-augmented generation (RAG) with enterprise data, making it well-suited for your RAG use case.

The first part of code initializes the PDF loader, loads the Ecommerce Categories, and splits it into chunks. You can obtain the Ecommerce Categories PDF referenced in this case study by clicking here. Subsequently, it initializes the OCI Gen AI LLM from the LangChain library.


from langchain_community.document_loaders import PyPDFLoader
from langchain_community.llms import OCIGenAI

loader = PyPDFLoader("/langflow/EcommerceCategories.pdf")
pages = loader.load_and_split()

llm = OCIGenAI(
    model_id="cohere.command",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="COMPARTMENT_OCID",
    model_kwargs={"temperature": 0.7, "top_p": 0.75, "max_tokens": 1000}
)

The next part of code initializes the OCI Gen AI embedding models and import prompt template, output parser and vector store libraries from the LangChain. It then stores the chunks of the category document into Chroma DB using the OCI embeddings model and creates a vector store.


from langchain_core.prompts import PromptTemplate
from langchain.schema.output_parser import StrOutputParser
from langchain.schema.runnable import RunnablePassthrough
from langchain_community.embeddings import OCIGenAIEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OCIGenAIEmbeddings(
    model_id="cohere.embed-english-light-v3.0",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="COMPARTMENT_OCID",
)
vectorstore = Chroma.from_documents(
    pages,
    embedding=embeddings    
)
retriever = vectorstore.as_retriever()

Finally, it sets the prompt and creates the chain by employing expression language to retrieve the relevant category from the vector store based on the product input. This retrieved category is then passed as context in prompt to LLM to classify the input product into the category.


template = """You are helpful assistant. Answer the question based only on the following context:
{context}
You will be given a product name from a user, you need to predict the category for given product name.
The response should only return the category name.
Product Name : {input}
"""
prompt = PromptTemplate.from_template(template)

chain = (
    {"context": retriever, "input": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)
print(chain.invoke("Adult Badminton Racket BR 580 Lite Blue Grey"))
Output: Sports and Outdoor

3. Purchase Order Details Agent

This use case demonstrates the functionality of agents in retrieving purchase order details for a specified purchase order using OCI Generative AI LLM. Agents within LangChain are engineered to interact with real-world scenarios, serving as robust tools for task automation. Agents utilize a blend of LLM, tools, and toolkit components to execute a predefined sequence of steps toward a specific objective.

LangChain recently introduced a new AI library called LangGraph, which streamlines the creation and management of AI agents and their runtimes. LangGraph introduces the capability to incorporate cycles, allowing for more intricate, agent-like behaviours. This enables the looping of an LLM to determine the next action to take.

This use case focuses on a straightforward scenario of retrieving purchase order details using LangGraph. Additionally, LangGraph enables users to define complex scenarios by incorporating multiple agents, each executing core tasks, along with a supervisor agent responsible for routing various task requests to these agents to fulfil a given objective.

The first part of following code initializes the OCI Gen AI LLM from the LangChain library. It then generates the tool from Python code, which invokes the Fusion purchase order REST service to fetch the status and amount of purchase order for the specified purchase order.


from langchain_community.llms import OCIGenAI

llm = OCIGenAI(
    model_id="cohere.command",
    service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",
    compartment_id="COMPARTMENT_OCID",
    model_kwargs={"temperature": 0.7, "top_p": 0.75, "max_tokens": 2000}
)

from langchain.tools import BaseTool, StructuredTool, Tool, tool

import requests
from requests.auth import HTTPBasicAuth
@tool("order_details", return_direct=True)
def order_details(PurchaseOrder: str) -> str:
    """useful when you need to get the purchase order details."""
    auth = HTTPBasicAuth('username', 'password')
    response = requests.get(f"https://fa-xxx-zzzz.fa.ocs.oraclecloud.com/fscmRestApi/resources/11.13.18.05/purchaseOrders?q=OrderNumber={PurchaseOrder}&fields=Status,Total,CurrencyCode&onlyData=true",auth=auth)
    if response.status_code >= 200 and response.status_code < 300:
        response = response.json()
        items = (response['items'])        
        amount = str(items[0]['Total']) + " "+ items[0]['CurrencyCode']
        return f"Status: {items[0]['Status']}, Amount: {amount}"
    else:
        return f"Failed to call events API with status code {response.status_code}"

tools = [order_details]

The next segment of the code retrieves the prompt from the LangChain hub designed for LLMs to manage JSON-specific input. Subsequently, it imports agent-specific libraries from LangChain and constructs an agent using a combination of LLMs, tools, and prompts. Additionally, it initializes the agent state, which will be continuously updated as the agent processes user input. The agent manages the conversation history, determines the agent outcome (Action/Finish), and handles intermediate steps, such as deciding to call the order_details tool to retrieve purchase order details. Additionally, it defines the tool executor to execute the tools.


from langchain import hub
prompt = hub.pull("hwchase17/react-chat-json")

from langchain.agents import create_json_chat_agent
from langchain_core.agents import AgentAction,AgentFinish
from typing import TypedDict, Annotated, List, Union
from langchain_core.messages import BaseMessage
import operator

agent = create_json_chat_agent(llm, tools, prompt)

class AgentState(TypedDict):
   input: str
   chat_history: list[BaseMessage]
   agent_outcome: Union[AgentAction, AgentFinish, None]
   intermediate_steps: Annotated[list[tuple[AgentAction, str]], operator.add]

from langgraph.prebuilt.tool_executor import ToolExecutor
tool_executor = ToolExecutor(tools)

The final segement of code defines three methods for invoking the agent, executing tools, and determining whether processing should continue or conclude. It then constructs the graph, a fundamental component of LangGraph, and incorporates agent and action nodes into it. The agent is set as the entry point, and conditional and normal edges are added to regulate the flow of execution in the agent execution process.

Subsequently, it receives input from the user, and the flow proceeds to the agent node, resulting in the invocation of the order_details tool and writing into the agent state. The flow then transitions to the action node, which executes the tool, records the results into the agent state as intermediate steps, and returns to the agent. The agent evaluates the agent state with the LLM, determines that it has the required response for the user input, concludes the process, and sends the purchase order details back to the user.


def run_agent(data):
    agent_outcome = agent.invoke(data)
    return {"agent_outcome": agent_outcome}

def execute_tools(data):
    agent_action = data['agent_outcome']
    output = tool_executor.invoke(agent_action)
    return {"intermediate_steps": [(agent_action, str(output))]}

def should_continue(data):
    if isinstance(data['agent_outcome'], AgentFinish):
        return "end"
    else:
        return "continue"

from langgraph.graph import END, StateGraph

workflow = StateGraph(AgentState)
workflow.add_node("agent", run_agent)
workflow.add_node("action", execute_tools)
workflow.set_entry_point("agent")

workflow.add_conditional_edges(
    "agent",
    {
        "continue": "action",
        "end": END
    }
)

workflow.add_edge('action', 'agent')
app = workflow.compile()

inputs = {"input": "Get the details of purchase order number 55116",
          "chat_history": [],
          "intermediate_steps":[]}
output = app.invoke(inputs)
print(output.get("agent_outcome").return_values['output'])

Output: Based on the purchase order number provided, it seems that the status is open and the amount is 3024.0 GBP.

The use cases are implemented using the LangChain Python library, although there is also a JavaScript library available, known as LangChain.js. The code can be deployed as Oracle Functions and accessed through an API gateway, as detailed in this blog post.

Conclusion

This blog post aims to outline various use cases for constructing applications with OCI Generative AI large language models and LangChain integration. However, it’s important to note that this is just the tip of iceberg, when developing production LLM applications, additional measures such as validations, guardrails, caching, monitoring, and evaluation of LLM responses are essential.

The authentication process for OCI Gen AI LLM mirrors that of other OCI services, with four authentication methods detailed in this document. For the above use cases, I’ve utilized the API key-based authentication method.

Extending SaaS by AI/ML features – Part 8: OCI Generative AI Integration with LangChain Use Cases

Introduction

Use Cases

1. Item Description Generation using AI

2. Item Classification using Retrieval Augmented Generation

3. Purchase Order Details Agent

Conclusion

Dipak Chhablani

Principal Solutions Architect | A-Team - Cloud Solution Architects

OCI Redundant IPSec Connections for GCP Connectivity

OCI JCA provider use case: Generate CSR

Extending SaaS by AI/ML features – Part 8: OCI Generative AI Integration with LangChain Use Cases

Introduction

Use Cases

1. Item Description Generation using AI

2. Item Classification using Retrieval Augmented Generation

3. Purchase Order Details Agent

Conclusion

Authors

Dipak Chhablani

Principal Solutions Architect | A-Team - Cloud Solution Architects

OCI Redundant IPSec Connections for GCP Connectivity

OCI JCA provider use case: Generate CSR