Asking about Events

LLM Vision can remember events and store them so you can ask about them later.

Asking Assist about remembered events

Adding Events to the Timeline

The easiest way to automatically add events to the timeline, is to use the official blueprint. In your own automations, set remember to true. If you want to add custom events, use the remember action.

The Conversation Integration

This guide has been tested with OpenAI Conversation, but should work with Google Generative AI and Anthropic Conversation as well.

To be able to ask about events in the timeline, you need to install and set up one of the integrations mentioned above.

Allow the Conversation Integration access to Assist

In the conversation integration settings you will need to set 'Control Home Assistant' to 'Assist'. This allows the integration to control devices exposed to assist.

Creating the Script

Why do I need to create this script?

Exposing calendar entities to Assist only exposes their current state (on or off) and the title of the next event. Since all events on the timeline are in the past, a script is needed to return the events.

1. Create a new script with the code below.

2. Expose the script to assist. Follow this guide to learn how to expose entities to assist.

3. Set up an assist pipeline in Home Assistant:

   1. Go to Settings > Voice Assistants

   2. 'Add Assistant'

   3. Give it a name and select your conversation integration as the 'Conversation Agent'

   4. (optional) If you want to be able to talk to your Assistant, add a TTS and STT configuration

This script will return events from the past day. To return more, change timedelta(days=1) to e.g. timedelta(days=7).

Note that processing many events can lead to high token usage.

script.get_camera_events

sequence:
  - variables:
      result:
        events: >-
          {% set titles = state_attr("calendar.llm_vision_timeline","events")%}
          {% set times = state_attr("calendar.llm_vision_timeline","starts") %}
          {% set summaries = state_attr("calendar.llm_vision_timeline","summaries") %}
          ```csv
          time, title, summary
          {%for i in range(titles|count)%}
          {%if times[i] > now() - timedelta(days=1)%}
          {{times[i]}}, '{{titles[i]}}', '{{summaries[i]}}'
          {%endif%}
          {%endfor%}
          ```
  - stop: stop
    response_variable: result
alias: Get Camera Events
description: Returns all events captured by cameras today.
icon: mdi:timeline
fields: {}

You can now ask Assist about events that happened today.

Troubleshooting

If Assist cannot answer questions about events, you may need to include some additional prompting in the system prompt. Instruct it to 'use the get_camera_events tool when asked about events'.