voice
Wake words and Home Assistant Assist on Android. The device listens on its own and talks to Assist like a Voice PE.
On this page
The voice side of an ESPHome voice assistant on
Android, as Home Assistant's Voice PE does it. The microphone, an on-device wake word
(microWakeWord, and openWakeWord models already on the
device as a fallback) on TensorFlow Lite, and the Assist client over
esphome-api's VoiceAssistantHandler: the run, the audio stream, the
response, announcements, continued conversations and voice timers.
It has no Pane API and no S1 classes (#152). The Glassworks assist pane wraps it with settings,
entities, the overlay and the speaker. Package dev.glassworks.voice, MIT licensed, except
MicroFrontend.kt, a port under Apache-2.0 and BSD-3-Clause (NOTICE). The AAR carries all
three texts (META-INF/LICENSE, LICENSE-APACHE-2.0, LICENSE-BSD-3-CLAUSE) and the NOTICE, and the
POM lists all three. minSdk 23.
Use it in your own app
val engine = VoiceEngine(context, VoiceConfig(), WakeWords.DEFAULT, onActiveWordChanged = { id -> /* store it, pass it next time */ })
val controller = VoiceAssistantController(
send = { type, payload -> server.sendVoice(type, payload) },
player = player, // plays the response and announcements: your media player
resolveUrl = { url -> if (url.startsWith("/")) "http://homeassistant.local:8123$url" else url }, // HA may send paths
wakeWords = engine, // Home Assistant's wake word select picks from these
emit = { event -> println(event) }, // WakeDetected, Listening, Thinking, Speaking, Idle, ...
)
engine.controller = controller
server.voice = controller // Home Assistant runs Assist through the ESPHome API
engine.start(muted = false)
// and about once a second: controller.tick()
server is an esphome-api ApiServer that Home Assistant has added. player is your UrlPlayer:
it plays the response and announcements from their URLs, one after another. Ask for RECORD_AUDIO
at runtime before start. stop the engine and close the controller when voice is turned off.
VoiceInput.onMicLevel gives each streamed frame's level (0 to 1), for an orb or a meter.
controller.cancel() ends a run (a tap on the overlay). The wake word models are fetched on first use
from esphome/micro-wake-word-models at a pinned commit, each checked against its SHA-256
(WakeWords.MICRO), into VoiceConfig.modelsDir.
What's in it
VoiceAssistantController |
The Assist client: Home Assistant's VoiceAssistantRequest, Audio, Event, Announce and Timer messages, as esphome's voice_assistant.cpp handles them for a device with a media player. Watchdogs end a run that stalls. Pure JVM. |
VoiceEngine, VoiceInput, VoiceConfig |
The microphone (16 kHz mono, 80 ms frames) and the wake word, feeding the controller. Loads a new word while the old one still listens, and retries a model that fails to load. |
WakeWords, WakeWord, MicroModel, WakeWordCatalog |
The official microWakeWord models with their default cutoffs, and how a stored word id reads. |
UrlPlayer, VoiceEvent |
What the app supplies and what it hears. |
| internal | The microWakeWord frontend (a port of TFLite Micro's) and detector, openWakeWord's feature pipeline, the TFLite glue and the model fetcher. |
The manifest adds RECORD_AUDIO and INTERNET. Logs use the tag S1Voice. How the S1 uses it, and
how the wake word was measured, is in docs/esphome.md and
docs/research/microwakeword.md.
Gradle
On Maven Central as dev.glassworks:voice (changelog). The first release, 0.1.0, is
not out yet: until it is, use the module from this repository (implementation(project(":voice"))).
repositories { mavenCentral() }
dependencies {
implementation("dev.glassworks:voice:0.1.0")
}
An Android library (AAR), minSdk 23 (Android 6). It brings in dev.glassworks:esphome-api, whose
VoiceAssistantHandler it implements, and TensorFlow Lite 2.16.1 (with its native library for every
ABI). A Kotlin build needs Kotlin 2.0 or newer.
Tests
./gradlew :voice:testDebugUnitTest. MicroFrontendTest compares the frontend with pymicro-features
on fixtures. VoiceAssistantControllerTest drives whole runs, announcements and timers.
VoiceEngineSwitchTest switches words under Robolectric. MicroWakeWordClipTest runs the four real
models (LiteRT) behind the Kotlin pipeline over a clip; it skips unless MWW_PYTHON points at a Python
with mww-test-requirements.txt installed, and CI's wake-word job runs it.