Google media events

dialogflow
google-assistant
#1

According to the Google API doc, when the Google Assistant API starts a MediaResponse, it can optionally indicate that it wants events from the built-in controls – specifically, it can ask to be told when the audio is FINISHED, PAUSED, STOPPED, or has FAILED.

The Jovo podcast player example’s google handler handles the FINISHED event, as ‘GoogleAction.Finished’. It does not handle the others. Attempting to use that as a template and implement the others did not seem to work.

I understand that Google “helpfully” handles a lot of the media control internally, but as the example demonstrates with FINISHED, sometimes our code will want to know what has happened.

So: Does Jovo support the other events? If so, what’s needed to enable them? If not, it seems this shouldn’t be hard to add to Jovo…

#2

Hey @keshlam

Could you provide a request object for the other events?

#3

This should work:


AUDIOPLAYER: {
		'GoogleAction.Paused'() {
			const progress = this.$googleAction!.$audioPlayer!.getProgress();
			// this will close the session
			this.tell('Playback paused');
		},
		'GoogleAction.Stopped'() {
			const progress = this.$googleAction!.$audioPlayer!.getProgress();
			// no response possible
		},
		'GoogleAction.Finished'() {
			this.tell('Playback finished');
		},
		'GoogleAction.Failed'() {
			this.tell('Playback failed');
		},
	}

}
#4

Are the exclamation points typos? I don’t recognize the syntax, and Javascript doesn’t seem to either. (Also, this being Google, shouldn’t that be .ask() rather than .tell(), to keep the session active so we get other events?)

If I remove them, the code is accepted – but I’m not getting the tell responses, or the console.log() output I also added, so there’s something not being connected properly. I’m also not seeing any output from Jovo itself indicating that it heard these events.

“jovo version” says I’m running 3.2.1

#5

My bad, I copied from the Typescript examples.

How do you test your action? I had issues with PAUSED and STOPPED in the simulator. Try it on a device.

Could you also provide the response json where you use playAudio?

#6

Yes, I’m testing on a Google Home device. (Trying to use the simulator with the Amazon version quickly convinced me that wasn’t complete enough for this project.)

Test sequence is simple enough:
“Hey Google, ask New Sounds On Demand to play the most recent episode.”
“Hey Google, pause.” (Or stop.)

My most recent attempt with Pause was apparently passed down to my Jovo code as a DateIntent, which currently returns an error; the device then says “New Sounds On Demand has stopped responding, please try again soon” but does pause the audio. So it seems Google Assistant heard me correctly but Jovo somehow didn’t.

Saying “Hey Google, resume” followed by another “Hey Google, pause” results in the pause not appearing at all on the Jovo console, though playback was stopped. Again, it seems something is getting lost between the system response and the Jovo event.

Console log for that whole sequence follows. By the way, is there anything in these logs that I should be masking before posting here, for security purposes? I want to give you everything you need, but not anything you don’t.

 >>>>> Request - 2021-10-26T18:35:21.028Z
{
   "responseId": "e9072031-aa51-465f-ae80-ef7f7aa0d1fe-7ef0c984",
   "queryResult": {
      "queryText": "play the most recent episode",
      "parameters": {},
      "allRequiredParamsPresent": true,
      "outputContexts": [
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_audio_output"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_media_response_audio"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_account_linking"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/google_assistant_input_type_voice"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/google_assistant_welcome"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/__system_counters__",
            "parameters": {
               "no-input": 0,
               "no-match": 0
            }
         }
      ],
      "intent": {
         "name": "projects/newsoundsondemand-jewn/agent/intents/ffa4adfb-e94a-4176-bd0e-5e930b11415f",
         "displayName": "LatestEpisodeIntent"
      },
      "intentDetectionConfidence": 1,
      "languageCode": "en"
   },
   "originalDetectIntentRequest": {
      "source": "google",
      "version": "2",
      "payload": {
         "user": {
            "locale": "en-US",
            "userStorage": "{\"userId\":\"c7d640f8-990a-43a0-ab74-6befdefdc667\"}",
            "userVerificationStatus": "VERIFIED"
         },
         "conversation": {
            "conversationId": "ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg",
            "type": "NEW"
         },
         "inputs": [
            {
               "intent": "actions.intent.MAIN",
               "rawInputs": [
                  {
                     "inputType": "VOICE",
                     "query": "ask new sounds on demand to play the most recent episode"
                  }
               ],
               "arguments": [
                  {
                     "name": "trigger_query",
                     "rawText": "play the most recent episode",
                     "textValue": "play the most recent episode"
                  }
               ]
            }
         ],
         "surface": {
            "capabilities": [
               {
                  "name": "actions.capability.AUDIO_OUTPUT"
               },
               {
                  "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
               },
               {
                  "name": "actions.capability.ACCOUNT_LINKING"
               }
            ]
         },
         "isInSandbox": true,
         "availableSurfaces": [
            {
               "capabilities": [
                  {
                     "name": "actions.capability.AUDIO_OUTPUT"
                  },
                  {
                     "name": "actions.capability.SCREEN_OUTPUT"
                  },
                  {
                     "name": "actions.capability.WEB_BROWSER"
                  }
               ]
            }
         ]
      }
   },
   "session": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg"
}

Checking server for new episodes...
Scan found known episode  4554 with known date 2021-10-25T04:00:00.000Z
Stopping incremental database update.
Highest numbered: #4554, Hybrid Choral Music
Most recent daily: #4409: Music Bridging Conflict and Cultures at 2021-10-26T04:00:00.000Z

 <<<<< Response - 2021-10-26T18:35:22.309Z
{
   "fulfillmentText": "Fetching episode #4409: Music Bridging Conflict and Cultures.",
   "end_interaction": false,
   "outputContexts": [],
   "payload": {
      "google": {
         "expectUserResponse": true,
         "richResponse": {
            "items": [
               {
                  "simpleResponse": {
                     "ssml": "<speak>Fetching episode #4409: Music Bridging Conflict and Cultures.</speak>"
                  }
               },
               {
                  "mediaResponse": {
                     "mediaType": "AUDIO",
                     "mediaObjects": [
                        {
                           "name": "#4409: Music Bridging Conflict and Cultures",
                           "contentUrl": "https://pdst.fm/e/www.podtrac.com/pts/redirect.mp3/audio.wnyc.org/newsounds/[email protected]&nyprBrowserId=NewSoundsOnDemand.smartspeaker.player"
                        }
                     ]
                  }
               }
            ]
         },
         "noInputPrompts": [
            {
               "ssml": "<speak>Fetching episode #4409: Music Bridging Conflict and Cultures.</speak>"
            }
         ],
         "userStorage": "{\"userId\":\"c7d640f8-990a-43a0-ab74-6befdefdc667\"}"
      }
   }
}

 >>>>> Request - 2021-10-26T18:35:38.568Z
{
   "responseId": "c8204204-0640-4af1-985f-ba193601e6e2-7ef0c984",
   "queryResult": {
      "queryText": "new sounds",
      "parameters": {
         "date": ""
      },
      "allRequiredParamsPresent": true,
      "outputContexts": [
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_audio_output",
            "parameters": {
               "date": "",
               "date.original": ""
            }
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_account_linking",
            "parameters": {
               "date": "",
               "date.original": ""
            }
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_media_response_audio",
            "parameters": {
               "date": "",
               "date.original": ""
            }
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/google_assistant_input_type_voice",
            "parameters": {
               "date": "",
               "date.original": ""
            }
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/__system_counters__",
            "parameters": {
               "no-input": 0,
               "no-match": 0,
               "date": "",
               "date.original": ""
            }
         }
      ],
      "intent": {
         "name": "projects/newsoundsondemand-jewn/agent/intents/d99d8f54-e352-495c-a616-184d6acc98f2",
         "displayName": "DateIntent"
      },
      "intentDetectionConfidence": 0.42478594,
      "languageCode": "en"
   },
   "originalDetectIntentRequest": {
      "source": "google",
      "version": "2",
      "payload": {
         "user": {
            "locale": "en-US",
            "userStorage": "{\"userId\":\"c7d640f8-990a-43a0-ab74-6befdefdc667\"}",
            "userVerificationStatus": "VERIFIED"
         },
         "conversation": {
            "conversationId": "ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg",
            "type": "ACTIVE",
            "conversationToken": "[]"
         },
         "inputs": [
            {
               "intent": "actions.intent.TEXT",
               "rawInputs": [
                  {
                     "inputType": "VOICE",
                     "query": "new sounds"
                  }
               ],
               "arguments": [
                  {
                     "name": "text",
                     "rawText": "new sounds",
                     "textValue": "new sounds"
                  }
               ]
            }
         ],
         "surface": {
            "capabilities": [
               {
                  "name": "actions.capability.AUDIO_OUTPUT"
               },
               {
                  "name": "actions.capability.ACCOUNT_LINKING"
               },
               {
                  "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
               }
            ]
         },
         "isInSandbox": true,
         "availableSurfaces": [
            {
               "capabilities": [
                  {
                     "name": "actions.capability.AUDIO_OUTPUT"
                  },
                  {
                     "name": "actions.capability.SCREEN_OUTPUT"
                  },
                  {
                     "name": "actions.capability.WEB_BROWSER"
                  }
               ]
            }
         ]
      }
   },
   "session": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg"
}

DateIntent: { name: '', value: '', key: '', id: '' }
DateIntent:
Checking server for new episodes...
Scan found known episode  4409 with known date 2021-10-26T04:00:00.000Z
Stopping incremental database update.
Highest numbered: #4554, Hybrid Choral Music
Most recent daily: #4409: Music Bridging Conflict and Cultures at 2021-10-26T04:00:00.000Z

  Error -----------------------------------------------------------------

  Message:
  Invalid time value

  Stack:
  RangeError: Invalid time value
      at format (C:\Users\keshlam\jovo\new-sounds-on-demand\node_modules\date-fns\format\index.js:392:11)
      at GoogleAction.DateIntent (C:\Users\keshlam\jovo\new-sounds-on-demand\src\app.js:507:60)
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at Function.applyHandle (C:\Users\keshlam\jovo\new-sounds-on-demand\node_modules\jovo-core\src\plugins\Handler.ts:179:24)
      at handle (C:\Users\keshlam\jovo\new-sounds-on-demand\node_modules\jovo-core\src\plugins\Handler.ts:241:5)
      at Middleware.run (C:\Users\keshlam\jovo\new-sounds-on-demand\node_modules\jovo-core\src\core\Middleware.ts:85:11)
      at App.handle (C:\Users\keshlam\jovo\new-sounds-on-demand\node_modules\jovo-core\src\core\BaseApp.ts:274:7)
      at App.handle (C:\Users\keshlam\jovo\new-sounds-on-demand\node_modules\jovo-framework\src\App.ts:333:5)
      at C:\Users\keshlam\jovo\new-sounds-on-demand\src\index.js:20:9

   ----------------------------------------------------------------------


  Request details:
  this.$googleAction initialized
  this.$type: {"type":"INTENT"}
  this.$session.$data : {}
  this.$nlu : {"intent":{"name":"DateIntent"}}
  this.$inputs : {"date":{"name":"","value":"","key":"","id":""}}


   ----------------------------------------------------------------------


 >>>>> Request - 2021-10-26T18:37:37.432Z
{
   "responseId": "64416b42-6076-4b0e-860e-83ae9c5941fa-7ef0c984",
   "queryResult": {
      "queryText": "resume",
      "parameters": {},
      "allRequiredParamsPresent": true,
      "outputContexts": [
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_audio_output"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_media_response_audio"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/actions_capability_account_linking"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/google_assistant_input_type_voice"
         },
         {
            "name": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg/contexts/__system_counters__",
            "parameters": {
               "no-input": 0,
               "no-match": 0
            }
         }
      ],
      "intent": {
         "name": "projects/newsoundsondemand-jewn/agent/intents/24ede82f-a364-47eb-9df3-01dea42ff8c7",
         "displayName": "ResumeIntent"
      },
      "intentDetectionConfidence": 0.800576,
      "languageCode": "en"
   },
   "originalDetectIntentRequest": {
      "source": "google",
      "version": "2",
      "payload": {
         "user": {
            "locale": "en-US",
            "userStorage": "{\"userId\":\"c7d640f8-990a-43a0-ab74-6befdefdc667\"}",
            "userVerificationStatus": "VERIFIED"
         },
         "conversation": {
            "conversationId": "ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg",
            "type": "ACTIVE",
            "conversationToken": "[]"
         },
         "inputs": [
            {
               "intent": "actions.intent.TEXT",
               "rawInputs": [
                  {
                     "inputType": "VOICE",
                     "query": "resume"
                  }
               ],
               "arguments": [
                  {
                     "name": "text",
                     "rawText": "resume",
                     "textValue": "resume"
                  }
               ]
            }
         ],
         "surface": {
            "capabilities": [
               {
                  "name": "actions.capability.AUDIO_OUTPUT"
               },
               {
                  "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
               },
               {
                  "name": "actions.capability.ACCOUNT_LINKING"
               }
            ]
         },
         "isInSandbox": true,
         "availableSurfaces": [
            {
               "capabilities": [
                  {
                     "name": "actions.capability.AUDIO_OUTPUT"
                  },
                  {
                     "name": "actions.capability.SCREEN_OUTPUT"
                  },
                  {
                     "name": "actions.capability.WEB_BROWSER"
                  }
               ]
            }
         ]
      }
   },
   "session": "projects/newsoundsondemand-jewn/agent/sessions/ABwppHG5LAd4sjiuvlPjz0nfvIU3Vof-wJTRBo7BrV8AfFnrsiBXjWY7p6AMNMFsCkBj9eIp_Yvjy2Ku94WDm_N3_BSbAg"
}

GOOGLE: Resume, https://pdst.fm/e/www.podtrac.com/pts/redirect.mp3/audio.wnyc.org/newsounds/[email protected]&nyprBrowserId=NewSoundsOnDemand.smartspeaker.player

 <<<<< Response - 2021-10-26T18:37:37.447Z
{
   "fulfillmentText": "Loading and resuming episode #4409: Music Bridging Conflict and Cultures.",
   "end_interaction": false,
   "outputContexts": [],
   "payload": {
      "google": {
         "expectUserResponse": true,
         "richResponse": {
            "items": [
               {
                  "simpleResponse": {
                     "ssml": "<speak>Loading and resuming episode #4409: Music Bridging Conflict and Cultures.</speak>"
                  }
               },
               {
                  "mediaResponse": {
                     "mediaType": "AUDIO",
                     "mediaObjects": [
                        {
                           "name": "#4409: Music Bridging Conflict and Cultures",
                           "contentUrl": "https://pdst.fm/e/www.podtrac.com/pts/redirect.mp3/audio.wnyc.org/newsounds/[email protected]&nyprBrowserId=NewSoundsOnDemand.smartspeaker.player"
                        }
                     ]
                  }
               }
            ]
         },
         "noInputPrompts": [
            {
               "ssml": "<speak>Loading and resuming episode #4409: Music Bridging Conflict and Cultures.</speak>"
            }
         ],
         "userStorage": "{\"userId\":\"c7d640f8-990a-43a0-ab74-6befdefdc667\"}"
      }
   }
}

Current code is checked into my Github project if you need to see exactly what I’ve written.

#7

May have found an unintended .tell(), which we’re warned has the Google side effect of closing the session so we won’t get these progress events. I’ll try again with that fixed and see if it behaves any better. (It would be nice if the Jovo coder didn’t have to be aware of this side effect difference, though I’m not sure how it could be swept under the rug cleanly unless the Google interface automagically turns all .tell()s into .ask()s… which may not be right for other applications.)

… Nope. Still nothing on the Jovo events console when I ask Google Assistant to pause the audio, or stop it.

#8

oh, I didn’t know you were using the legacy version with Dialogflow.

The audioplayer events are very limited there: https://developers.google.com/assistant/df-asdk/reference/webhook/rest/Shared.Types/Status

#9

Ah. Since I used the Jovo example code as a starting point, and it is (or was) still written in terms of Dialogflow (though there’s a note telling us that the newer solution exists)…OK, I’ll have to look at migrating across.

Along with migrating to lambda, v4 of Jovo, and Typescript… (Sigh.) Well, at least I have the prototype running.