PaLM 2 token_counts Result

[aliacar]

Hi all,

I need help to understand the result of count_tokens method in Vertex AI Python SDK

from vertexai.preview.language_models import TextGenerationModel

parameters = {
    "temperature": 0.2,
    "max_output_tokens": 200,
    "top_p": .8,
    "top_k": 40,
}
model = TextGenerationModel.from_pretrained("text-bison@001")

PROMPT = "Write a weekend plan for a family of three."v
response = model.predict(
    PROMPT,
    **parameters,
)
token_count = model.count_tokens(PROMPT)

 The above code gives the response as expected. However, I did not understand the result of count_tokens method. The below is the result for given text, PROMPT.

CountTokensResponse(total_tokens=55, total_billable_characters=35, _count_tokens_response=total_tokens: 55
total_billable_characters: 35
)

The total_billable_characters makes sense, but I did not understand why total_tokens is 55 when there is a total of 43 characters in the given text PROMPT.

Can anyone explain what a token means in the result above?

Thanks.

[nceniza · Accepted solution]

This reference might be helpful to your inquiry:

https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/CountTokensResponse

"The total number of tokens counted across all instances from the request."

[aliacar]

Hey @nceniza. Thanks for your answer.

[omar1990]

great question.. but i still dont get why its 55

[aliacar]

Let me explain it with an example:

from vertexai.preview.language_models import TextGenerationModel

model_name = "text-bison@001"

parameters = {
"temperature": 0.2,
"max_output_tokens": 200,
"top_p": .8,
"top_k": 40,
}

model = TextGenerationModel.from_pretrained(model_name)

prompt = "Write a weekend plan for a family of three."

print(model.count_tokens(prompt))
print('----')
print(model.count_tokens([prompt]))

Output:

CountTokensResponse(total_tokens=55, total_billable_characters=35, _count_tokens_response=total_tokens: 55
total_billable_characters: 35
)
----
CountTokensResponse(total_tokens=10, total_billable_characters=35, _count_tokens_response=total_tokens: 10
total_billable_characters: 35
)

In the version I use, the input of the `count_tokens` function should be a List of string. Otherwise, I think it miscounts tokens by iterating the string input char by char.

So, using List of string instead of string is my solution to this issue.

[omar1990]

thank you so much; can you please also explain to me the difference between 

vertexai.preview.language_models

and 

vertexai.language_models

[aliacar]

Sorry, I don't know the difference.