Skip to main content
POST
/
api
/
v2
/
tokenize
Create text tokens
curl --request POST \
  --url https://paradigm.lighton.ai/api/v2/tokenize \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "model": "<string>",
  "prompt": "<string>",
  "messages": [
    {}
  ]
}
'
{
  "total_tokens": 123,
  "request_model": "<string>",
  "model_used": "<string>",
  "tokenizer_type": "<string>",
  "id": "<string>",
  "model": "<string>"
}

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

Request serializer for tokenize endpoint.

model
string
required

Model to use for tokenization, must exist and be configured from the admin

prompt
string

The text to tokenize

messages
object[]

List of messages to tokenize (alternative to prompt)

Response

200 - application/json

Response serializer for tokenize endpoint results.

total_tokens
integer
required

Total number of tokens in the input text

request_model
string
required

The requested model for tokenization

model_used
string
required

The model used for tokenization

tokenizer_type
string
required

The type of tokenizer used

id
string
required

The id of the response

model
string
required

The model used for tokenization