Install models

This guide provides instructions on how to configure KubeAI models.

Installing models with helm

KubeAI provides a chart that contains preconfigured models.

Preconfigured models with helm

When you are defining Helm values for the kubeai/models chart you can install a preconfigured Model by setting enabled: true. You can view a list of all preconfigured models in the chart's default values file.

# helm-values.yaml
catalog:
  llama-3.1-8b-instruct-fp8-l4:
    enabled: true

You can optionally override preconfigured settings, for example, resourceProfile:

# helm-values.yaml
catalog:
  llama-3.1-8b-instruct-fp8-l4:
    enabled: true
    resourceProfile: nvidia-gpu-l4:2 # Require "2 NVIDIA L4 GPUs"

Custom models with helm

If you prefer to add a custom model via the same Helm chart you use for installed KubeAI, you can add your custom model entry into the .catalog array of your existing values file for the kubeai/models Helm chart:

# helm-values.yaml
catalog:
  my-custom-model-name:
    enabled: true
    features: ["TextEmbedding"]
    owner: me
    url: "hf://me/my-custom-model"
    resourceProfile: CPU:1

Installing models with kubectl

You can add your own model by defining a Model yaml file and applying it using kubectl apply -f model.yaml.

If you have a running cluster with KubeAI installed you can inspect the schema for a Model using kubectl explain:

kubectl explain models
kubectl explain models.spec
kubectl explain models.spec.engine

Feedback welcome: A model management UI

We are considering adding a UI for managing models in a running KubeAI instance. Give the GitHub Issue a thumbs up if you would be interested in this feature.