---
title: "Language model inference | Tarka Docs"
description: "Create OpenAI-compatible Chat Completions and Responses with Tarka language models."
url: "https://tarkahq.com/docs/inference/llms/"
---

Available Inference · Language models

# Language models

Use OpenAI Chat Completions or Responses with a Tarka base URL and a model alias returned by model discovery.

Provisioning and management **Use the console or the Control API.**

Manage inference keys, model access, and organization limits in the console or through the Control API. Chat requests themselves go to the Inference API.

[Open console](https://tarkahq.com/console) [Control API reference](https://tarkahq.com/docs/control-api/)

Quickstart

## Create a chat completion

Create a server-side key with the `inference` scope and set it as `TARKA_API_KEY`.

Python `pip install openai`

```
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://tarka.rest/v1",
    api_key=os.environ["TARKA_API_KEY"],
)

completion = client.chat.completions.create(
    model="himalaya-q8",
    messages=[
        {"role": "user", "content": "Reply with READY."},
    ],
)

print(completion.choices[0].message.content)
```

Shell `POST /chat/completions`

```
curl "https://tarka.rest/v1/chat/completions" \
  -H "Authorization: Bearer $TARKA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "himalaya-q8",
    "messages": [
      {"role": "user", "content": "Reply with READY."}
    ]
  }'
```

Model discovery

## Available language-model aliases

A key can call only the aliases assigned to it. Query model discovery at runtime and use the returned`id` as the `model` value.

Shell `GET /v1/models`

```
curl "https://tarka.rest/v1/models" \
  -H "Authorization: Bearer $TARKA_API_KEY"
```

| Family | Alias | Precision | Choose it for |
| --- | --- | --- | --- |
| **GLM-5.3 Flash** | `glm-5.3-flash` | — | High-volume chat, extraction, and classification |
| **GLM-5.3 Flash** | `glm-5.3-flash-eu` | — | European provider endpoint |
| **HimalayaGPT 0.5B** | `himalaya-q4` | Q4 K M | Smallest footprint |
| **HimalayaGPT 0.5B** | `himalaya-q8` | Q8 | Balanced default |
| **HimalayaGPT 0.5B** | `himalaya-bf16` | BF16 | Highest-precision variant |
| **HimalayaGPT 0.5B** | `himalaya-q4-private` | Q4 K M | Private · no internal retention |
| **HimalayaGPT 0.5B** | `himalaya-q8-private` | Q8 | Private · no internal retention |
| **HimalayaGPT 0.5B** | `himalaya-bf16-private` | BF16 | Private · no internal retention |
| **DeepSeek V4.1 Flash** | `deepseek-v4.1-flash` | — | Fast general-purpose chat, tool use, and coding |
| **DeepSeek V4.1 Flash** | `deepseek-v4.1-flash-eu` | — | European provider endpoint |
| **Himalaya Gemma 4 E2B** | `himalaya-gemma-4-q4` | Q4 K M | Smallest footprint |
| **Himalaya Gemma 4 E2B** | `himalaya-gemma-4-q8` | Q8 | Balanced default |
| **Himalaya Gemma 4 E2B** | `himalaya-gemma-4-bf16` | BF16 | Highest-precision variant |
| **Himalaya Gemma 4 E2B** | `himalaya-gemma-4-q4-private` | Q4 K M | Private · no internal retention |
| **Himalaya Gemma 4 E2B** | `himalaya-gemma-4-q8-private` | Q8 | Private · no internal retention |
| **Himalaya Gemma 4 E2B** | `himalaya-gemma-4-bf16-private` | BF16 | Private · no internal retention |
| **Kimi K3** | `kimi-k3` | — | Coding agents and long, multi-step tasks |
| **GLM 5.3** | `glm-5.3` | — | Chat and tool calls through the US provider endpoint |
| **Qwen 3.8 Flash Next** | `qwen3.8-flash-next` | — | Free chat and tool calls |

Model access can be narrower for an organization or individual key. If an alias does not appear in`GET /v1/models`, create or update the key in the console or through the Control API before sending inference requests.

Endpoint reference

## POST /chat/completions

Send an ordered conversation and receive an OpenAI-shaped completion or server-sent event stream. Generate clients and tooling from the public[Tarka inference OpenAPI contract](https://github.com/TarkaHQ/api/blob/main/openapi/tarka-inference-v1.openapi.json).

`model` string · required

An alias returned by `GET /models`.

`messages` array · required

Ordered system, user, assistant, and supported tool messages.

`stream` boolean · default false

Return SSE chunks when `true`.

`max_tokens` integer · optional

Maximum generated tokens, bounded by the model.

`temperature` number · optional

Sampling randomness. Support and range are model-specific.

`top_p` number · optional

Nucleus sampling threshold.

`stop` string or array

One or more generation stop sequences.

`seed` integer · optional

Requests repeatable sampling where supported.

`tools` array · optional

Tool definitions for aliases that advertise tool support.

`user` string · optional

Your end-user identifier for abuse and audit correlation.

### Request

JSON `request body`

```
{
  "model": "himalaya-q8",
  "messages": [
    {"role": "system", "content": "Answer concisely."},
    {"role": "user", "content": "Why is the sky blue?"}
  ],
  "temperature": 0.2,
  "max_tokens": 256,
  "stream": false
}
```

### Response

JSON `200 OK`

```
{
  "id": "chatcmpl_01K3...",
  "object": "chat.completion",
  "created": 1787596800,
  "model": "himalaya-q8",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Sunlight is scattered by molecules in the atmosphere..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 24,
    "completion_tokens": 31,
    "total_tokens": 55
  }
}
```

**Unknown OpenAI fields are preserved for compatible providers.**

Model capabilities still vary. Discover the alias first and do not assume that every model supports tools, images, structured output, or every sampling option.

Unified generation

## POST /responses

Send text or message input and receive an OpenAI-shaped Response. Stored Responses can be retrieved, chained with `previous_response_id`, cancelled while active, and deleted.

### OpenAI Python client

Python `client.responses.create`

```
response = client.responses.create(
    model="himalaya-q8",
    input="Reply with READY.",
)

print(response.output_text)
```

### HTTP request

Shell `POST /v1/responses`

```
curl "https://tarka.rest/v1/responses" \
  -H "Authorization: Bearer $TARKA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "himalaya-q8",
    "instructions": "Answer concisely.",
    "input": "Why is the sky blue?",
    "max_output_tokens": 256,
    "store": true
  }'
```

`model` string · required

An alias returned by `GET /models`.

`input` string or input-item array · required

Text, messages, function calls, or function-call outputs.

`instructions` string · optional

Developer instructions prepended to the input.

`stream` boolean · default false

Return typed Responses API events as SSE.

`store` boolean · default false

Explicitly retain tenant-scoped response state for 30 days or until deletion. Existing clients using previous\_response\_id must send store: true. This is independent of bucket capture.

`previous_response_id` string · optional

Continue from a stored response owned by the same organization.

`tools` array · optional

Function and custom tool definitions. Hosted OpenAI tools are not accepted.

`max_output_tokens` integer · optional

Maximum generated output tokens.

### Stored response lifecycle

Shell `retrieve · cancel · delete`

```
curl "https://tarka.rest/v1/responses/resp_0123456789abcdef" \
  -H "Authorization: Bearer $TARKA_API_KEY"

curl -X POST "https://tarka.rest/v1/responses/resp_0123456789abcdef/cancel" \
  -H "Authorization: Bearer $TARKA_API_KEY"

curl -X DELETE "https://tarka.rest/v1/responses/resp_0123456789abcdef" \
  -H "Authorization: Bearer $TARKA_API_KEY"
```

- Streaming emits typed events such as `response.created`, `response.output_text.delta`, and `response.completed`.
- `background: true`, hosted tools, Conversations, and reusable prompt templates are not part of the supported contract.
- Deletion follows the official OpenAI Python client contract and returns no SDK object.

Native gRPC

## Call the same operations with protobuf

Use `tarka.inference.v2.InferenceService` at `grpc.tarka.rest:443`. The package name remains `v2` for wire compatibility; canonical REST is `/v1`.

grpcurl `CreateResponse`

```
grpcurl \
  -H "authorization: Bearer $TARKA_API_KEY" \
  -d '{"body":{"model":"himalaya-q8","input":"Reply with READY."}}' \
  grpc.tarka.rest:443 \
  tarka.inference.v2.InferenceService/CreateResponse
```

Language-model RPCs include raw OpenAI Chat Completions and Responses unary/stream methods, typed Chat Completions methods, model discovery, and stored Response retrieve/delete/cancel methods.

Streaming

## Consume server-sent events

Set `stream: true`. Each event contains a `chat.completion.chunk` and the stream ends with `data: [DONE]`.

Python `OpenAI SDK`

```
stream = client.chat.completions.create(
    model="himalaya-q8",
    messages=[{"role": "user", "content": "Write one short sentence."}],
    stream=True,
)

for chunk in stream:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
```

Wire format `text/event-stream`

```
data: {"id":"chatcmpl_01K3...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl_01K3...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl_01K3...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
```

- Do not assume every chunk contains text; role, finish, or usage chunks may not.
- Treat `[DONE]` as the clean end of the stream.
- If the connection closes before `[DONE]`, treat the generation as incomplete.

## Data retention

See the [inference capture guide](https://tarkahq.com/docs/control-api/inference-capture/) for console setup, REST and MCP examples, stored formats, and delivery health.

Free Himalaya aliases retain request inputs and outputs in a Tarka-owned archive for evaluation and debugging until Tarka administrators delete them. They are not used for model training.

Aliases ending in `-private` have no Tarka internal retention. They use the `inference` key scope and are currently priced at zero. Other model families also have internal retention off.

Organization owners and admins can separately enable capture in [Object Storage](https://tarkahq.com/console/storage). All model calls then write JSON records and media into that organization’s `inference-history` bucket. Captures remain until deleted by default, with optional 7-, 30-, or 90-day expiry. Changing the setting affects new requests; disabling does not delete existing objects.

Storage failures do not stop inference. Delivery status shows capture gaps. Download captures through S3; permanent deletion in versioned buckets must remove historical versions as well as the current object.

Uploaded audio and inline images are saved as binary objects beside each request’s JSON record. Remote media URLs are saved as sanitized references; their remote content is not copied. Large requests and interrupted deliveries can also leave capture gaps.

The Responses API’s `store` flag controls only its separate conversation state. It now defaults to `false`. Send `store: true` to retrieve a response later or use it with `previous_response_id`; stored state expires after 30 days. This flag does not override free-Himalaya internal retention or organization bucket capture.