Using On-Premises LLM with Spotfire Copilot: Configuration and Deployment

[Naoki Nakamichi]

Hello,

I am looking for guidance on how to configure and deploy an on-premises LLM (e.g., Llama3) with Spotfire Copilot. Specifically, I have two main questions:

1. How should the .env file be configured if we want to use an on-premises LLM instead of a cloud-based LLM like OpenAI?
2. What are the necessary steps and requirements to deploy the on-premises LLM as a service to integrate it with Copilot?

Any detailed instructions, best practices, or examples would be greatly appreciated.

Thank you!

[Prashant Maske]

Hello Naoki,

First of all you, will need to run the model locally and then make it accessible as a web service that Spotfire Copilot can communicated with. Ensure that your Spotfire Copilot instance is configured to point to the LLM service URL specified in the .env file.

Thanks

[Naoki Nakamichi]

Hi Prashant,

Thank you very much for your help. I will try using LocalAI for this. I appreciate your cooperation.

Best regards,

 

[Markku Mikkola]

Hi,

I'm trying also to setup and experiment the spotfire-copilot backend with on-prem LLMs (e.g. LLaMA3.1-8b..) using an API endpoint compatible with OpenAI's API request and response structures.

Now I have this issue with Milvus "data-loader" when loading the Spotfire documents, that even if I point my "AZURE_OPENAI_ENDPOINT" to the on-prem API endpoint still the "data-loader" makes  request towards "https://api.openai.com/v1/embeddings" which result "401 Unauthorized" as it tries to authenticate with my on-prem API key.

Also Im wondering the Description for "AZURE_OPENAI_ENDPOINT" variable in the documentation:

"Defines the endpoint URL for the Azure OpenAI API, enabling users to set the API endpoint in the system and establish the connection to Azure OpenAI service. Not required if not using Azure OpenAI"

So how do I correctly configure data-loader with onprem API endpoint when not using Azure OpenAI API?

Thanks!

[Marcelo Gallardo]

 

Hello Markku,

Thank you for reaching out! We’re happy to assist with your inquiries regarding Spotfire Copilot.

First, could you let us know which version of Copilot you’re using? The ability to configure local models was introduced in later releases, so this will help us guide you accordingly.

Additionally, please share the interface your local model uses. We currently support the Ollama interface for Llama models, but if you're using a different interface, we can assist in providing the necessary plugin. Please note that we support all models that are compatible with the Langchain library.

Regarding the data loaders, if you’re using a local embedding model, some configuration will be required to properly integrate it with the data loader. We can help walk you through this setup if needed.

Let us know if this answers your question, or if you’d prefer, we can schedule a call to discuss further and provide more detailed guidance.

Best regards,
Marcelo Gallardo

[Markku Mikkola]

Hi Marcelo,

Thanks for you quick response.

I'm using Spotfire Copilot 1.1.0 package and instructions included in it with small deviation that I deployed the backend components on k8s.

If necessary to discuss about the interfaces used I think better to setup a call, but first if possible if you could help with the data loader configuration problem and I guess some additional configuration would be needed for the orchestrator as well?

BR,

Markku

 

[Naoki Nakamichi]

Hello Markku,

I'm experiencing the same issue. Ollama functions as an OpenAI API-compatible service, but it's generating an error related to the API key.

The error message is as follows:

ERROR: Exception in ASGI application Traceback (most recent call last): File "/app/main.py", line 208, in handle_request process_docs(docs=docs, vectordb=vector_db) File "/app/main.py", line 89, in process_docs raise e File "/app/main.py", line 85, in process_docs vectordb.add_documents(chuncked_docs) File "/usr/local/lib/python3.10/site-packages/langchain_core/vectorstores.py", line 138, in add_documents return self.add_texts(texts, metadatas, **kwargs) File "/usr/local/lib/python3.10/site-packages/langchain_community/vectorstores/milvus.py", line 558, in add_texts embeddings = self.embedding_func.embed_documents(texts) File "/usr/local/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 517, in embed_documents return self._get_len_safe_embeddings(texts, engine=engine) File "/usr/local/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 333, in getlen_safe_embeddings response = self.client.create( File "/usr/local/lib/python3.10/site-packages/openai/resources/embeddings.py", line 113, in create return self._post( File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1232, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 921, in request return self._request( File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1012, in _request raise self._make_status_error_from_response(err.response) from None openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: dummy. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

The version of Copilot I'm using is 1.1.0.

Best regards,

[Marcelo Gallardo]

Hello @Markku Mikkola.

 

Apologies for the ongoing difficulties with your setup.

Since you're working on an on-premise deployment that includes embedding models, could you please provide more details about the specific models you're using? Currently, the data loader is configured to assume the use of OpenAI's embedding model, which is why it’s attempting to connect to OpenAI.

Once we have the details of the models you're working with, I can share the correct configuration and URL for a data loader that supports your chosen embedding model.

If you'd prefer to discuss your environment further, feel free to schedule a call, and we'd be happy to assist you.

 

Best regards,

Marcelo Gallardo

[Markku Mikkola]

Hi Marcelo,

We have internally fine tuned embedding models and Allenai's specter-2 model: https://huggingface.co/allenai/specter2_base available via a text embedding API endpoint (onprem).

I tried to reach you on email where we can discuss more in details.

BR,

Markku

[Markku Mikkola]

Hi Marcelo,

Is there any general guideline and documentation available how to configure data-loaders/orchestrator with onprem embedding and LLM models and APIs and if possible how to develop and "extend" Copilot backend components with user environment specific plugins? I'm referring to the plugin support mentioned in "Getting Started with Spotfire Copilot" PDF document included in the Copilot 1.1.0 release.

BR,

Markku

[Naoki Nakamichi]

@Marcelo Gallardo

Hello Marcelo,

I've decided to use OpenAI's embeddings for the data loader for now.

Could you please explain the specific configuration method for the .env file of the orchestrator?

I'm using Ollama as the interface, set up to be usable as an OpenAI-compatible API. I'm using the Llama3 model. No API key is required.

Best regards,