Table of documentation contents

How to use Weaviate with your own vectors? (No modules, pure vector storage and search)

Learn how to use Weaviate with your own vectors (no modules).

Introduction

Weaviate can be used without any modules, as pure vector native database and search engine. If you choose not to include any modules, you will need to enter a vector for each data entry. You can then search through the objects by a vector as well. On this page you will learn how to set up Weaviate without modules, enter data objects with vectors and search through them. This tutorial uses a simple example to run Weaviate locally using docker-compose.

Prerequisites

1. Have docker-compose installed
Make sure to have "docker-compose" installed on your machine. Check out this guide for more info about docker-compose and Weaviate.

Step 1 - Set up a Weaviate instance without any module attached.

Download the Docker Compose configuration file.

$ curl -o docker-compose.yml "https://configuration.semi.technology/v2/docker-compose/docker-compose.yml?enterprise_usage_collector=false&media_type=none&runtime=docker-compose&weaviate_version=v1.7.1"

Start up Weaviate

You can now start up with the whole setup by running:

$ docker-compose up -d

Check if your Weaviate is running

Weaviate should now be running and exposed on your host port 8080. You can verify with docker ps or simply send a test query to Weaviate which should return something like the following:

$ curl localhost:8080/v1/schema

which should return {"classes":[]}.

Step 2 - Adding a data schema.

Even when you want to add data objects with a custom defined vector, you will need to define the objects’ structure in a data schema. Let’s create a simple schema for this example:

{
  "classes": [{
      "class": "Post",
      "vectorizer": "none", # explicitly tell Weaviate not to vectorize anything, we are providing the vectors ourselves
      "properties": [{
          "name": "content",
          "dataType": ["text"],
      }]
  }]
}

Add this to Weaviate using one of the clients or with a curl command:

  import weaviate

client = weaviate.Client("http://localhost:8080")

class_obj = {
  "classes": [{
    "class": "Post",
    "vectorizer": "none", # explicitly tell Weaviate not to vectorize anything, we are providing the vectors ourselves through our BERT model
    "properties": [{
        "name": "content",
        "dataType": ["text"],
    }]
  }]
}

client.schema.create_class(class_obj)

Step 3 - Import data to Weaviate

For this example, we’re uploading one data object to our Weaviate instance. If you want to add more objects at once, check how “batching” works.

We need to define the vector with the data object, because we don’t have a vectorization module in Weaviate that does this for us now. Add the complete vector as follows when uploading a data object:

  import weaviate

client = weaviate.Client("http://localhost:8080")

data_obj = {
    "content": "This is a news update of May 28 2021. Today ......."
}

client.data_object.create(
  data_obj, 
  "Post", 
  "36ddd591-2dee-4e7e-a3cc-eb86d30a0934",
  "vector": [0.3, 0.2, 0.1, .... 0.9], # supported types are `list`, 'numpy.ndarray`, `torch.Tensor` and `tf.Tensor`. Make sure the length matches with your Weaviate settings.
)

Check the API reference for more information around creating objects.

Step 4 - Query and search through data

You can search through vector data with Weaviate. If you don’t have any modules attached, you can enter a vector and Weaviate will return vectors that are closest to this vector (computed by the cosine distance). Make sure the length of the search vector is the same as the vector length of the objects in your Weaviate instance.

Seaching for and by vectors can be done with GraphQL. You can use an explore filter to find data objects close to a specified vector in your dataset. The nearVector{} filter is structured as follows for the Get{} function:

  {
  Get{
    Post(
      nearVector: {
        vector: [-0.36840257,0.13973749,-0.28994447,-0.18607682,0.20019795,0.15541431,-0.42353877,0.30262852,0.2724561,0.07069917,0.4877447,0.038771532,0.64523,-0.15907241,-0.3413626,-0.026682584,-0.63310874,-0.33411884,0.082939014,0.30305764,0.045918174,-0.21439327,-0.5005205,0.6210859,-0.2729049,-0.51221114,0.09680918,0.094923325,-0.15688285,-0.07325482,0.6588305,0.0523736,-0.14173415,-0.27428055,0.25526586,0.057506185,-0.3103442,0.028601522,0.124522656,0.66984487,0.12160647,-0.5090515,-0.540393,-0.39546522,-0.2201204,0.34625968,-0.21068871,0.21132985,0.048714135,0.09043683,0.3176081,-0.056684002,-0.12117501,-0.6591976,-0.26731065,0.42615625,0.33333477,-0.3240578,-0.18771006,0.2328068,-0.17239179,-0.33583146,-0.6556605,-0.10608161,-0.5135395,-0.25123677,-0.23004892,0.7036331,0.04456794,0.41253626,0.27872285,-0.28226635,0.11927197,-0.4677766,0.4343466,-0.17538455,0.10621233,0.95815116,0.23587844,-0.006406698,-0.10512518,-1.1125883,-0.37921682,0.040789194,0.676718,0.3369762,0.040712647,0.580487,0.20063736,-0.021220192,-0.09071747,-0.0023735985,0.30007777,-0.039925132,0.4035474,-0.2518212,-0.17846306,0.12371392,-0.0703354,-0.3752431,-0.652917,0.5952828,1.3426708,-0.08167235,-0.38515738,0.058423538,-0.08100355,-0.192886,0.3745164,-0.23291737,0.33326542,-0.6019264,-0.42822492,-0.6524583,-0.15210791,-0.5073593,0.022548754,-0.058033653,-0.47369233,-0.30890635,0.6338296,0.0017854869,0.1954949,0.99348027,-0.26558784,-0.058124136,1.149388,0.02915948,0.013422121,0.25484946,-0.030017598,-0.23879935,0.053123385,-0.36463016,-0.0024245526,0.1202083,-0.45966506,-0.34140104,-0.08484162,-0.03537422,-0.2817959,0.25044164,-0.5060605,0.1252808,-0.032539487,0.110069446,-0.20679846,-0.46421885,-0.4141739,0.26994973,-0.070687145,0.16862138,-0.20162229,0.22199251,-0.2771402,0.23653336,0.16585203,-0.08286354,-0.15343396,0.23893964,-0.7453282,-0.16549355,-0.1947069,0.46136436,0.22064126,0.28654936,-0.038697664,0.037633028,-0.80988157,0.5094175,-0.0920082,0.25405347,-0.64169943,0.43366328,-0.2999211,-0.4090591,0.11957859,0.00803617,-0.0433745,0.12818244,0.28464508,-0.31760025,0.16558012,-0.33553946,-0.3943465,0.59569097,-0.6524206,0.3683173,-0.60456693,0.2046492,0.46010277,0.24695799,0.2946015,0.11376746,-0.027988048,0.03749422,-0.16577742,0.23407385,-0.0231737,-0.023245076,0.08752677,0.2299883,0.35467404,0.046193745,-0.39828986,0.21079691,0.38396686,-0.0018698421,0.16047359,-0.057517264,-0.203534,0.23438136,-0.84250915,0.22371331,0.0058325706,0.30733636,0.19518353,-0.108008966,0.6509316,0.070131645,-0.24023099,0.28779706,0.2326336,0.07004021,-0.45955566,0.20426086,-0.37472793,-0.049604423,0.4537271,0.6133582,-1.0527759,-0.5472505,0.15193434,0.5296606,-0.11560251,0.07279209,0.40557706,0.2505283,0.24490519,0.017602902,-0.004647707,0.16608049,0.12576887,0.118216865,0.4403996,0.39552462,-0.22196701,-0.061155193,0.03693534,-0.4022908,0.3842317,-0.0831345,0.01930883,0.3446575,-0.2167439,-0.23994556,-0.09370326,-0.3671856,0.044011243,0.017895095,-0.019855855,-0.16416992,0.17858285,0.31287143,0.38368022,-0.006513525,0.45780763,-0.23027879,0.108570844,-0.4449492,-0.035763215,0.03818417,0.040017277,-0.17022872,-0.2622464,0.65610534,0.16720143,0.2515769,-0.23535803,0.62484455,0.16771325,-0.62404263,0.19176348,-0.72786695,0.18485649,-0.30914405,-0.3230534,-0.24064465,0.28841522,0.39792386,0.15618932,0.03928854,0.18277727,-0.101632096,0.1868196,-0.33366352,0.086561844,0.48557812,-0.6198209,-0.07978742]
      }
    ){
      content
      _additional {
        certainty
      }
    }
  }
}

More Resources

If you can’t find the answer to your question here, please look at the:

  1. Frequently Asked Questions. Or,
  2. Knowledge base of old issues. Or,
  3. For questions: Stackoverflow. Or,
  4. For issues: Github. Or,
  5. Ask your question in the Slack channel: Slack.
Tags
  • how to
  • no modules