Indexes are the “atom” of Vectorize. Vectors are inserted into an index and enable you to query the index for similar vectors for a given input vector.
Creating an index requires three inputs:
- A name, for example
- The (fixed) of each vector, for example 384 or 1536.
- The (fixed) to use for calculating vector similarity.
The configuration of an index cannot be changed after creation.
Create an index
To create an index with
$ npx wrangler vectorize create your-index-name --dimensions=NUM_DIMENSIONS --metric=SELECTED_METRIC
$ npx wrangler vectorize create your-index-name --dimensions=768 --metric=cosine
Dimensions are determined from the output size of the machine learning (ML) model used to generate them, and are a function of how the model encodes and describes features into a vector embedding.
The number of output dimensions can determine vector search accuracy, search performance (latency), and the overall size of the index. Smaller output dimensions can be faster to search across, which can be useful for user-facing applications. Larger output dimensions can provide more accurate search, especially over larger datasets and/or datasets with substantially similar inputs.
The number of dimensions an index is created for cannot change. Indexes expect to receive dense vectors with the same number of dimensions.
The following table highlights some example embeddings models and their output dimensions:
|Model / Embeddings API||Output dimensions||Use-case|
|Workers AI - ||768||Text|
|OpenAI - ||1536||Text|
|Cohere - ||768||Text|
|Google Cloud - ||1408||Multi-modal (text, images)|
Distance metrics are functions that determine how close vectors are from each other. Vectorize indexes support the following distance metrics:
|Distance is measured between |
|Euclidean (L2) distance. |
|Negative dot product. Larger negative values or smaller positive values denote more similar vectors. A score of |
Determining the similarity between vectors can be subjective based on how the machine-learning model that represents features in the resulting vector embeddings. For example, a score of
0.8511 when using a
cosine metric means that two vectors are close in distance, but whether data they represent is similar is a function of how well the model is able to represent the original content.
Distance metrics cannot be changed after index creation, and that each metric has a different scoring function.