Usage

Learn how to use LLM Vision

Available Actions

LLM Vision exposes five actions:

1. image_analyzer: Analyzes image files, image and camera entities

2. video_analyzer: Analyzes video files and Frigate events

3. stream_analyzer: Records and analyzes camera entities

4. data_analyzer: Updates a sensor value based on images

5. create_event: Add a custom event to the LLM Vision Timeline

Response

image, video, stream and data analyzers will return a response value:

title: White SUV seen
response_text: >-
  The image shows a white SUV parked on a driveway.
key_frame: /config/media/llmvision/snapshots/0b9ff287-0.jpg

Depending on your configuration, not all keys may be present. Refer to the parameter reference of each action to learn more.

Accessing the response

To access it, set response_variable in your automation or script to e.g. response.

To get e.g. the generated title use {{response.title}}.

Using Snapshots in your own Automations

In your own automations use the following template for snapshot paths:

{{ response.key_frame.replace('/media', '/media/local') }}

Where response is the response variable of an analyzer action with expose_images enabled.

Structured Responses

Since v1.6.0 LLM Vision supports structured responses (json) for some providers.

When response_format is enabled and structure is provided, some providers can generate a valid JSON object according to the specified JSON schema.

Structure Example

{
  "type": "object",
  "properties": {
    "title": {
      "type": "string",
      "description": "Event title"
    },
    "description": {
      "type": "string",
      "description": "Event description"
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 100
    }
  },
  "required": [
    "title",
    "description",
    "confidence"
  ],
  "additionalProperties": false
}

Schema Properties

Field	Type	Required	Allowed Values	Description
type	string	Yes	"object"	Specifies the JSON data type the schema describes. In this case, the schema validates a JSON object.
properties	object	Yes	Key-value pairs where each key is a field name and each value is a nested schema definition	Defines the allowed fields of the object and the validation rules for each field.
required	array of strings	No (but commonly used)	Array of property names defined in properties	Lists which properties must be present in the JSON object for it to be valid.
additionalProperties	boolean (or schema)	No	true, false, or a schema object	Controls whether properties not listed in properties are allowed. false means no extra fields are permitted.