AI Accelerator - Pipelines overview

As part of the EDB Postgres AI's AI Accelerator Pipelines abstracts away the complexity of working with AI data. It transforms Postgres into a powerful platform for AI data management, as it combines vector search from PGvector with automation for complex AI workflows.

Pipelines, Auto Embedding, and Intelligent Retriever

Pipelines handles the entire lifecycle of embedding generation, storage, and indexing — so you can simply select your desired model from our list of supported models, or connect to any OpenAI API compatible external model, and let Pipelines manage everything else from data ingestion to efficient similarity search.

With Auto Embedding, Pipelines automates embedding creation and updates, ensuring that vector stores remain up to date without manual embedding management — reducing the risk of stale data causing GenAI errors and hallucinations.

The Intelligent Retriever feature makes it possible to perform similarity and semantic searches across text and image data, whether stored in Postgres or object storage — all with a single retriever function call.

How Pipelines works

Pipelines delivers its functionality through the Pipelines aidb extension, embedded into the Postgres server. The extension provides a set of functions and views that allow you to interact with the AI data in your Postgres database.

Retrievers

Pipelines' aidb extension introduces the concept of a “retriever” that you can create for a given type and location of AI data. Currently, Pipelines supports unstructured plain text documents as well as a set of image formats. This data can either reside in regular columns of a Postgres table or it can reside in an S3-compatible object storage bucket.

A retriever encapsulates all processing that is needed to make the AI data in the provided source location searchable and retrievable through similarity. The application just needs to create a retriever via the aidb.register_retriever_for_table() function for Postgres tables or aidb.register_retriever_for_volume for externally stored data on S3 or local filesystems.

Auto embedding

Auto embedding is currently supported for AI data stored in Postgres tables and it automates the embedding updates using Postgres triggers. aidb.enable_auto_embedding_for_table() enables auto embedding for a given retriever and aidb.disable_auto_embedding_for_table() disables it.

By default, auto embedding is disabled and the application has to manually trigger embedding generation for all existing data in the source location using aidb.bulk_embedding().

You can make best use of the two options by bulk loading your exisiting data into your table with auto embedding disabled. Then you can run aidb.bulk_embedding() to create embeddings for that data. Following that you can enable auto embedding to create embeddings for any future data added to the table.

All embedding generation, storage, indexing, and management is handled by Pipelines' aidb extension internally. The application just has to specify the encoder LLM that the retriever should be using for the specific data and use case.

Intelligent retriever

Once a retriever is created and all embeddings are up to date, the application can just use aidb.retrieve_key() to run a similarity search and retrieval by providing a query input. It will return the id of the matched items. When the retriever is created for text data, the query input is also a text term. For image retrievers the query input is an image. If you are using a multi-modal model, such as CLIP, with your retriever, you can use text or images as source and query inputs. This allows you to search for images using text or vice versa.

The aidb retriever makes sure to use the same encoder LLM for the query input, conducts a similarity search and finally returns the ranked list of similar data from the source location. For Postgres tables, you can also use aidb.retrieve_text() to retrieve the actual data from the source table.

Pipelines currently supports a broad list of open encoder LLMs from HuggingFace as well as a set of OpenAI encoders. HuggingFace LLMs are run locally on the Postgres node, while OpenAI encoders involve a call out to the OpenAI cloud service. You can view the list of currently supported model providers by viewing the aidb.model_providers table.