Custom Providers and OpenAI-Compatible Endpoints

A custom provider is a named connection you add next to the built-in ones. Use one for:

• OpenAI-compatible servers: LM Studio, vLLM, llama.cpp server, LocalAI, a LiteLLM proxy, or your company's gateway.
• Other LiteLLM adapters: Azure OpenAI, Mistral, Cerebras and the other providers LiteLLM supports.

Add a custom provider in VS CodeCopy Anchor Link

1. Open Settings → API Keys & Connectors and select Add custom provider.
2. Fill in the form and select Save connection. Andromity saves it and fetches the endpoint's model list.
3. On the new card, select Test to send a short request, then Use model to make it the active model.

Each custom card has Use model, Edit, Test and Remove. Built-in connections can't be removed, and you can't remove the connection that is currently active; select another model first.

Add one in the terminal appCopy Anchor Link

Open Settings (Ctrl+E) → API Keys → Add custom provider. The form has the same fields, with Save & test and Save.

Add one in config.tomlCopy Anchor Link

A custom provider is a [[providers]] entry in config.toml:

[[providers]]
name = "local-models"
display_name = "Local models"
type = "openai"
base_url = "http://localhost:1234/v1"
model = "qwen2.5-coder-7b-instruct"
api_key = ""

[default]
provider = "local-models"
model = "qwen2.5-coder-7b-instruct"

api_key accepts a $VARIABLE reference, as for built-in connections. Andromity validates entries saved from the app; check hand-written ones against the rules below.

ExamplesCopy Anchor Link

LM StudioCopy Anchor Link

Start the local server in LM Studio, then:

[[providers]]
name = "lmstudio"
display_name = "LM Studio"
type = "openai"
base_url = "http://localhost:1234/v1"
model = "qwen2.5-coder-7b-instruct"

vLLM, llama.cpp server or LocalAICopy Anchor Link

[[providers]]
name = "vllm"
type = "openai"
base_url = "http://localhost:8000/v1"
model = "Qwen/Qwen2.5-Coder-32B-Instruct"

LiteLLM proxy or a company gatewayCopy Anchor Link

[[providers]]
name = "corp-llm"
display_name = "Company gateway"
type = "openai"
base_url = "https://llm-gateway.example.internal/v1"
model = "gpt-4.1"
api_key = "$CORP_LLM_TOKEN"

Azure OpenAICopy Anchor Link

Use your deployment name as the model ID:

[[providers]]
name = "azure"
type = "azure"
base_url = "https://YOUR-RESOURCE.openai.azure.com"
model = "YOUR-DEPLOYMENT"
api_version = "2024-10-21"
api_key = "$AZURE_OPENAI_KEY"

MistralCopy Anchor Link

Adapters other than openai don't need a base URL:

[[providers]]
name = "mistral"
type = "mistral"
model = "mistral-large-latest"
api_key = "$MISTRAL_API_KEY"

If you leave api_key empty for a non-openai adapter, LiteLLM looks for that provider's standard environment variable itself.

How requests are sentCopy Anchor Link

Andromity passes the connection to LiteLLM as <type>/<model>, with the base URL as api_base and the API version if set. For example, type = "openai" and model = "qwen2.5-coder" becomes openai/qwen2.5-coder. If the model ID already starts with <type>/, it is used unchanged.

When an openai-type custom connection has no key, Andromity sends the placeholder not-required. This stops LiteLLM from sending your OPENAI_API_KEY to an unrelated endpoint.

Andromity waits up to 10 minutes for the first token from Ollama or any endpoint on localhost, to allow time for a model to load, and 60 seconds for other endpoints.

Test a connectionCopy Anchor Link

Test (or Save & test in the terminal app) sends Reply with OK. and waits up to 15 seconds for the first token. No workspace content is sent. Common results:

Model list, context window and pricingCopy Anchor Link

For openai and anthropic types with a base URL, Andromity calls GET <base URL>/models (following pages) when you save the connection or select Refresh Catalog in the Model Hub. The models appear under the connection's chip in the Hub.

• Context window: taken from the endpoint's context_length, context_window or max_input_tokens field. If the endpoint reports none, Andromity assumes 131,072 tokens. There is no setting to override it; context compaction and the context indicator use this number.
• Pricing: shown when the endpoint returns pricing.prompt and pricing.completion. Otherwise the Hub shows Pricing unavailable, and usage costs fall back to LiteLLM's price list or show as an estimate of ~$0.0000.

Local models with OllamaCopy Anchor Link

Ollama is built in, so it doesn't need a custom provider. Andromity lists the models you have installed (from /api/tags), reads each model's context length from /api/show, and passes it to Ollama as num_ctx. Ollama usage always costs $0.

To use an Ollama server that isn't on http://localhost:11434, set its address in any of these places:

• VS Code: the andromity.ollamaHost setting, in your user settings. Andromity saves it to config.toml when it changes and each time the engine starts.
• Terminal app: Settings → Model → Ollama Base URL.
• By hand: config.toml:

[[providers]]
name = "ollama"
type = "ollama"
base_url = "http://192.168.1.20:11434"

All three set the same address. If andromity.ollamaHost is set, it wins when VS Code starts. A workspace's .vscode/settings.json can't set it, so a repository can't send your code to a different server. See Local AI Coding with Ollama for setup and model choice.