Video Analyzer

Example

Minimal working example

action: llmvision.video_analyzer
data:
  provider: 01JAAFDSVEJEBMESBP62QP156T # Pick in UI
  message: Describe what happens in this video frame by frame.
  max_frames: 5
  max_tokens: 1000
  video_file: |-
    /config/www/tmp/front_door.mp4
  include_filename: false

Action Reference

You can call video_analyzer to analyze either local video files using the video_file or a Frigate event using event_id. You must pass at least one of the above parameters:

Action Reference

action: llmvision.video_analyzer
data:
  provider: 01J99F4T99PA1XGQ4CTQS3CP8H  # Select in UI dropdown
  message: What is happening in the video?
  max_tokens: 1000
  model: gpt-4o-mini
  video_file: |-
    /config/www/tmp/front_door.mp4
    /config/www/tmp/garage.mp4
  event_id: 1712108310.968815-r28cdt
  max_frames: 5 # Analyze the 5 most relevant frames
  target_width: 1280
  include_filename: true

The provider id will not be the same for you. Switch to UI mode and select one of your configurations. If you don't see any, you need to set up at least one provider!

For all available models see: Choosing the right model

Parameter Reference

Parameter	Required	Description	Default
provider	Yes	The AI provider configuration
model	No	Model used for processing the image(s).
message	Yes	The prompt to send along with the image(s).
remember	No	Remember the analyzed event	false
use_memory	No	Use information stored in memory to provide additional context. Memory must be set up.	false
video_file	No*	The path to the video file(s). Each path must be on a new line.
event_id	No*	Event ID from Frigate. Each id must be on a new line.
frigate_retry_attempts	No	How many times to retry fetching the video clip from Frigate. Clips are not always available from Frigate as soon as the event has ended. Slower machines or longer clips may need additional attempts. Increase this if you see errors fetching the clips from Frigate in your automation traces.
frigate_retry_seconds	No	How long to wait between retries to fetch the video clip from Frigate. Clips are not always available from Frigate as soon as the event has ended. Slower machines or longer clips may need additional attempts. Increase this if you see errors fetching the clips from Frigate in your automation traces.
max_frames	No	How many frames to analyze. Will pick the most relevant frames (most motion)	3
include_filename	Yes	Whether to include the filename in the request.	false
target_width	No	Width to downscale the image to before encoding.	1280
max_tokens	Yes	The maximum number of response tokens to generate.	1000
generate_title	No	Generate and returns a title in the response. (Used for notifications and remembered events)	false
expose_images	No	Save key frame to /config/media/llmvision/snapshots. File path is included in response as key_frame. If used together with remember images will be deleted after retention_time set in Timeline. Otherwise this folder will use a lot of disk space!	false