Generative AI overview
This document describes the generative artificial intelligence (AI) features
that BigQuery ML supports. These features let you perform AI tasks in
BigQuery ML by using pre-trained Vertex AI
foundation models
.
Supported tasks include the following:
You access a Vertex AI model to perform one of these functions by
creating a
remote model
in BigQuery ML that represents the Vertex AI
model's endpoint. Once you have created a remote model over the
Vertex AI model that you want to use, you access that model's
capabilities by running a BigQuery ML function against the
remote model.
This approach lets you use the capabilities of these Vertex AI
models to analyze BigQuery data by using SQL.
Workflow
You can use
remote models over Vertex AI models
and
remote models over Cloud AI services
together with BigQuery ML functions in order to accomplish
complex data analysis and generative AI tasks.
The following diagram shows some typical workflows where you might use these
capabilities together:
Generative AI
You can use large language models (LLMs) to perform tasks such as text
summarization and generation. For example, you could summarize a long report, or
analyze sentiment in customer feedback. You can use vision language models
(VLMs) to analyze visual content such as images and videos for tasks
like visual captioning and visual Q&A. You can use multimodal models to perform
the same tasks as LLMs and VLMs, plus additional tasks like audio transcription
and document analysis.
To perform generative AI tasks, you can create
a reference to a pre-trained Vertex AI
model
by creating a remote model and specifying the model name
for the
ENDPOINT
value. The following Vertex AI models
are supported:
gemini-1.5-flash
(
Preview
)
gemini-1.5-pro
(
Preview
)
gemini-1.0-pro
gemini-1.0-pro-vision
(
Preview
)
text-bison
text-bison-32k
text-unicorn
When you create a remote model that references any version of a
Vertex AI
text-bison
model, you can optionally choose
to configure
supervised tuning
(
Preview
) at the same time.
After you create the model, you can use the
ML.GENERATE_TEXT
function
to interact with that model:
- For remote models based on
gemini-1.0-pro
,
text-bison
,
text-bison-32k
,
or
text-unicorn
models, you can use the
ML.GENERATE_TEXT
function with
a prompt you provide in a query or from a column in a
standard table
.
- For remote models based on the
gemini-1.0-pro-vision
model, you can use the
ML.GENERATE_TEXT
function to analyze image or video content from an
object table
with a prompt you
provide as a function argument.
- For remote models based on the
gemini-1.5-flash
or
gemini-1.5-pro
models, you can use the
ML.GENERATE_TEXT
function to analyze text, image,
audio, video, or PDF content from an object table with a prompt you
provide as a function argument, or you can generate text from a prompt you
provide in a query or from a column in a standard table.
You can use
grounding
and
safety attributes
when you use Gemini models with the
ML.GENERATE_TEXT
function, provided that
you are using a standard table for input. Grounding lets the
Gemini model use additional information from the internet to
generate more specific and factual responses. Safety attributes let the
Gemini model filter the responses it returns based on the
attributes you specify.
All inference occurs in Vertex AI. The results are stored in
BigQuery.
To learn more, try
generating text with the
ML.GENERATE_TEXT
function
.
Embedding
You can use embedding to identify semantically similar items. For example, you
can use text embedding to identify how similar two pieces of text are. If
the pieces of text are semantically similar, then their respective embeddings
are located near each other in the embedding vector space.
You can use BigQuery ML models to create the following types of
embeddings:
- To create text embeddings, you can create a reference to one of the
Vertex AI
text-embedding
or
text-multilingual-embedding
embedding models
by creating a remote model and specifying the LLM name for the
ENDPOINT
value.
- To create multimodal embeddings, which can embed text, images, and videos into
the same semantic space, you can create a reference to the Vertex AI
multimodalembedding
LLM by creating a remote model and specifying the
LLM name for the
ENDPOINT
value. This feature is in
preview
.
- To create embeddings for structured
independent and identically distributed random variables (IID)
data, you can use a
Principal component analysis (PCA)
model or an
Autoencoder
model. This feature is in
preview
.
- To create embeddings for user or item data, you can use a
Matrix factorization
model. This feature is in
preview
.
After you create the model, you can use the
ML.GENERATE_EMBEDDING
function
to interact with it. For all types of supported models,
ML.GENERATE_EMBEDDING
works with data in
standard tables
. For multimodal
embedding models,
ML.GENERATE_EMBEDDING
also works with visual
content in
object tables
. For remote
models, all inference occurs in Vertex AI. For other model types,
all inference occurs in BigQuery. The results are stored in
BigQuery.
To learn more, try creating
text embeddings
and
image embeddings
with the
ML.GENERATE_EMBEDDING
function.
For a smaller, lightweight text embedding, try
using a pretrained
TensorFlow model
,
such as NNLM, SWIVEL, or BERT.
For information about choosing the best model for your embedding use case, see
Choosing a text embedding model
.
What's next