NextIntent goes to GoogleAction.Finished

google-assistant

#1

I have a podcast player app with the AUDIOPLAYER state. inside the AUDIOPLAYER state i have implemented GoogleAction.Finished
Also the app has Next and Previous intents implemented.
On Alexa devices this works just fine. But on google devices it routes to GoogleAction.Finished sometimes and sometines it executes the NextIntent.
Is this the expected behavior or I’m missing something?


#2

I’m afraid it’s an ASR/NLU issue that is handled by the Google Assistant.

Could you provide the request json? What’s the understood text? (in queryText)


#3

Hello Alex. Nothing is in querytext.
When it routes to next intent in queryText theres next. When it routes to GoogleAction.Finished theres nothing on the queryText. Also the Media Status is Finished. And it seems that the request is the same when I let a podcast naturally finish on its own. This is an issue because I need to execute different things on skipping an audio file and letting it finish on its own.
Also another issue that happens is that previous has the same erratic behavior. Sometimes it goes to previousIntent sometimes (majority of times) it goes to the GoogleAction.Finished directive.
Here are the requests:
Route to GoogleAcion.Finished when you say next or previous:

    {
    "responseId": "c6f4c539-8ed7-4edb-9e4f-81baa0b8b413-4639978a",
    "queryResult": {
        "parameters": {},
        "allRequiredParamsPresent": true,
        "fulfillmentText": "I didn't get that. Can you repeat?",
        "fulfillmentMessages": [
            {
                "text": {
                    "text": [
                        "I didn't get that. Can you repeat?"
                    ]
                }
            }
        ],
        "outputContexts": [
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ/contexts/actions_capability_media_response_audio"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ/contexts/actions_capability_web_browser"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ/contexts/actions_capability_audio_output"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ/contexts/actions_capability_screen_output"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ/contexts/actions_capability_account_linking"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ/contexts/__system_counters__",
                "lifespanCount": 1,
                "parameters": {
                    "no-input": 0,
                    "no-match": 2
                }
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ/contexts/actions_intent_media_status",
                "parameters": {
                    "MEDIA_STATUS": {
                        "@type": "type.googleapis.com/google.actions.v2.MediaStatus",
                        "status": "FINISHED"
                    }
                }
            }
        ],
        "intent": {
            "name": "projects/dgts-ms/agent/intents/29b19b1e-2b9f-41a3-837a-e2f3a10b8760",
            "displayName": "Default Fallback Intent",
            "isFallback": true
        },
        "intentDetectionConfidence": 1,
        "languageCode": "en"
    },
    "originalDetectIntentRequest": {
        "source": "google",
        "version": "2",
        "payload": {
            "user": {
                "locale": "en-US",
                "lastSeen": "2021-06-30T19:44:28Z",
                "userStorage": "{\"userId\":\"d6eb62ee-0cef-43c8-820c-2e2ac0ae75d3\"}",
                "userVerificationStatus": "VERIFIED"
            },
            "conversation": {
                "conversationId": "ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ",
                "type": "ACTIVE",
                "conversationToken": "[\"__system_counters__\"]"
            },
            "inputs": [
                {
                    "intent": "actions.intent.MEDIA_STATUS",
                    "rawInputs": [
                        {}
                    ],
                    "arguments": [
                        {
                            "name": "MEDIA_STATUS",
                            "extension": {
                                "@type": "type.googleapis.com/google.actions.v2.MediaStatus",
                                "status": "FINISHED"
                            }
                        }
                    ]
                }
            ],
            "surface": {
                "capabilities": [
                    {
                        "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
                    },
                    {
                        "name": "actions.capability.WEB_BROWSER"
                    },
                    {
                        "name": "actions.capability.AUDIO_OUTPUT"
                    },
                    {
                        "name": "actions.capability.SCREEN_OUTPUT"
                    },
                    {
                        "name": "actions.capability.ACCOUNT_LINKING"
                    }
                ]
            },
            "isInSandbox": true,
            "availableSurfaces": [
                {
                    "capabilities": [
                        {
                            "name": "actions.capability.AUDIO_OUTPUT"
                        },
                        {
                            "name": "actions.capability.SCREEN_OUTPUT"
                        },
                        {
                            "name": "actions.capability.WEB_BROWSER"
                        }
                    ]
                }
            ]
        }
    },
    "session": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHGmmDTwwcxAteodlWHSnYonlSiFWuo97txEhCnkg4MTe08O1ieJtxODvywpqguyICR5RQ"
}

This is the request when it actually routes to NextIntent:

 {
    "responseId": "6bac81ae-1673-4470-97cc-f24e5a571910-4639978a",
    "queryResult": {
        "queryText": "Next",
        "parameters": {},
        "allRequiredParamsPresent": true,
        "fulfillmentMessages": [
            {
                "text": {
                    "text": [
                        ""
                    ]
                }
            }
        ],
        "outputContexts": [
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw/contexts/actions_capability_audio_output"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw/contexts/actions_capability_account_linking"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw/contexts/actions_capability_screen_output"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw/contexts/actions_capability_media_response_audio"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw/contexts/actions_capability_web_browser"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw/contexts/google_assistant_input_type_touch"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw/contexts/__system_counters__",
                "lifespanCount": 1,
                "parameters": {
                    "no-input": 0,
                    "no-match": 0
                }
            }
        ],
        "intent": {
            "name": "projects/dgts-ms/agent/intents/f86b71a2-fce9-4ef3-ad4e-34e4a7732106",
            "displayName": "NextIntent"
        },
        "intentDetectionConfidence": 1,
        "languageCode": "en"
    },
    "originalDetectIntentRequest": {
        "source": "google",
        "version": "2",
        "payload": {
            "user": {
                "locale": "en-US",
                "lastSeen": "2021-07-01T14:28:58Z",
                "userStorage": "{\"userId\":\"d6eb62ee-0cef-43c8-820c-2e2ac0ae75d3\"}",
                "userVerificationStatus": "VERIFIED"
            },
            "conversation": {
                "conversationId": "ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw",
                "type": "ACTIVE",
                "conversationToken": "[\"__system_counters__\"]"
            },
            "inputs": [
                {
                    "intent": "actions.intent.TEXT",
                    "rawInputs": [
                        {
                            "inputType": "TOUCH",
                            "query": "Next"
                        }
                    ],
                    "arguments": [
                        {
                            "name": "text",
                            "rawText": "Next",
                            "textValue": "Next"
                        }
                    ]
                }
            ],
            "surface": {
                "capabilities": [
                    {
                        "name": "actions.capability.AUDIO_OUTPUT"
                    },
                    {
                        "name": "actions.capability.ACCOUNT_LINKING"
                    },
                    {
                        "name": "actions.capability.SCREEN_OUTPUT"
                    },
                    {
                        "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
                    },
                    {
                        "name": "actions.capability.WEB_BROWSER"
                    }
                ]
            },
            "isInSandbox": true,
            "availableSurfaces": [
                {
                    "capabilities": [
                        {
                            "name": "actions.capability.AUDIO_OUTPUT"
                        },
                        {
                            "name": "actions.capability.SCREEN_OUTPUT"
                        },
                        {
                            "name": "actions.capability.WEB_BROWSER"
                        }
                    ]
                }
            ]
        }
    },
    "session": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHFykljl87vixQQ9Q-cn-q6KDncFyum15knJVTxVZv7qI_KP6SpHhU5nRoIHNNszCpZlBw"
}

This is the request when it routes to PreviousIntent

 {
    "responseId": "2004626e-3b07-4d51-a5f3-035324776b4d-4639978a",
    "queryResult": {
        "queryText": "previous",
        "parameters": {},
        "allRequiredParamsPresent": true,
        "fulfillmentMessages": [
            {
                "text": {
                    "text": [
                        ""
                    ]
                }
            }
        ],
        "outputContexts": [
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_audio_output"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_account_linking"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_screen_output"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_media_response_audio"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_web_browser"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/google_assistant_input_type_voice"
            },
            {
                "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/__system_counters__",
                "lifespanCount": 1,
                "parameters": {
                    "no-input": 0,
                    "no-match": 0
                }
            }
        ],
        "intent": {
            "name": "projects/dgts-ms/agent/intents/11570c88-05c5-4399-8a6e-cd21b7580d95",
            "displayName": "PreviousIntent"
        },
        "intentDetectionConfidence": 1,
        "languageCode": "en"
    },
    "originalDetectIntentRequest": {
        "source": "google",
        "version": "2",
        "payload": {
            "user": {
                "locale": "en-US",
                "lastSeen": "2021-07-01T14:32:31Z",
                "userStorage": "{\"userId\":\"d6eb62ee-0cef-43c8-820c-2e2ac0ae75d3\"}",
                "userVerificationStatus": "VERIFIED"
            },
            "conversation": {
                "conversationId": "ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg",
                "type": "ACTIVE",
                "conversationToken": "[\"__system_counters__\"]"
            },
            "inputs": [
                {
                    "intent": "actions.intent.TEXT",
                    "rawInputs": [
                        {
                            "inputType": "VOICE",
                            "query": "previous"
                        }
                    ],
                    "arguments": [
                        {
                            "name": "text",
                            "rawText": "previous",
                            "textValue": "previous"
                        }
                    ]
                }
            ],
            "surface": {
                "capabilities": [
                    {
                        "name": "actions.capability.AUDIO_OUTPUT"
                    },
                    {
                        "name": "actions.capability.ACCOUNT_LINKING"
                    },
                    {
                        "name": "actions.capability.SCREEN_OUTPUT"
                    },
                    {
                        "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
                    },
                    {
                        "name": "actions.capability.WEB_BROWSER"
                    }
                ]
            },
            "isInSandbox": true,
            "availableSurfaces": [
                {
                    "capabilities": [
                        {
                            "name": "actions.capability.WEB_BROWSER"
                        },
                        {
                            "name": "actions.capability.AUDIO_OUTPUT"
                        },
                        {
                            "name": "actions.capability.SCREEN_OUTPUT"
                        }
                    ]
                }
            ]
        }
    },
    "session": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg"
}

And finally the request when a podcast finished naturally on its own:

{
 "responseId": "04012886-979d-4756-be83-a8dda0ec335f-4639978a",
 "queryResult": {
     "parameters": {},
     "allRequiredParamsPresent": true,
     "fulfillmentText": "I didn't get that. Can you repeat?",
     "fulfillmentMessages": [
         {
             "text": {
                 "text": [
                     "I didn't get that. Can you repeat?"
                 ]
             }
         }
     ],
     "outputContexts": [
         {
             "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_audio_output"
         },
         {
             "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_web_browser"
         },
         {
             "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_media_response_audio"
         },
         {
             "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_account_linking"
         },
         {
             "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_capability_screen_output"
         },
         {
             "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/__system_counters__",
             "lifespanCount": 1,
             "parameters": {
                 "no-input": 0,
                 "no-match": 2
             }
         },
         {
             "name": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg/contexts/actions_intent_media_status",
             "parameters": {
                 "MEDIA_STATUS": {
                     "@type": "type.googleapis.com/google.actions.v2.MediaStatus",
                     "status": "FINISHED"
                 }
             }
         }
     ],
     "intent": {
         "name": "projects/dgts-ms/agent/intents/29b19b1e-2b9f-41a3-837a-e2f3a10b8760",
         "displayName": "Default Fallback Intent",
         "isFallback": true
     },
     "intentDetectionConfidence": 1,
     "languageCode": "en"
 },
 "originalDetectIntentRequest": {
     "source": "google",
     "version": "2",
     "payload": {
         "user": {
             "locale": "en-US",
             "lastSeen": "2021-07-01T14:33:23Z",
             "userStorage": "{\"userId\":\"d6eb62ee-0cef-43c8-820c-2e2ac0ae75d3\"}",
             "userVerificationStatus": "VERIFIED"
         },
         "conversation": {
             "conversationId": "ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg",
             "type": "ACTIVE",
             "conversationToken": "[\"__system_counters__\"]"
         },
         "inputs": [
             {
                 "intent": "actions.intent.MEDIA_STATUS",
                 "rawInputs": [
                     {}
                 ],
                 "arguments": [
                     {
                         "name": "MEDIA_STATUS",
                         "extension": {
                             "@type": "type.googleapis.com/google.actions.v2.MediaStatus",
                             "status": "FINISHED"
                         }
                     }
                 ]
             }
         ],
         "surface": {
             "capabilities": [
                 {
                     "name": "actions.capability.AUDIO_OUTPUT"
                 },
                 {
                     "name": "actions.capability.WEB_BROWSER"
                 },
                 {
                     "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
                 },
                 {
                     "name": "actions.capability.ACCOUNT_LINKING"
                 },
                 {
                     "name": "actions.capability.SCREEN_OUTPUT"
                 }
             ]
         },
         "isInSandbox": true,
         "availableSurfaces": [
             {
                 "capabilities": [
                     {
                         "name": "actions.capability.AUDIO_OUTPUT"
                     },
                     {
                         "name": "actions.capability.SCREEN_OUTPUT"
                     },
                     {
                         "name": "actions.capability.WEB_BROWSER"
                     }
                 ]
             }
         ]
     }
 },
 "session": "projects/dgts-ms/agent/environments/__aog-6/users/-/sessions/ABwppHG4ORAIuAEItUxVhvpjh-S7my6TEY1hPx11RZNskpI7WbwS4X5y1gEJmCOaev73nZ6XZg"
}

This has been tested on a mobile device, Nest hub, and speaker. The behavior is the same.


#4

Yes, looks like an issue of the platform itself. Not sure if there are any workarounds. I haven’t worked on Google Actions with the media responses besides the example project.

@dominik-meissner Do you have any best practices in your team?

btw. I edited your post for better readability. The content wasn’t edited.


#5

Lots of headaches with the Google mediaplayer…
This is a dialogflow project and we have completely moved to conversational actions so i am not 100% in the topic. But in general mediaplayer on Goolgle behaves totally different then on Alexa. Especially the previous/next functionality is (as of Conversational Actions) moved into system intents - even if you provide these intents in your language model. You never know when a user skips to the next song (you webhook is not triggered). Sometimes the MEDIA_STATUS_FINISHED is triggered on a skip but you cant distinguish between a end of song or a skip. We are in the middle of some investigations of a possible workaround (not using playlists anymore) and let you know once we know more. We would also like to see more effort from Google side to improve the mediaplayer.