Multi surface conversations

google-assistant

#1

Hi guys,

Is it possible to have multi surface conversations on the google device as explained in the link below?

https://developers.google.com/actions/assistant/surface-capabilities

I want to be able to start a conversation on google home, and continue on my phone. Is this possible?

Regards,
Afam


#2

Hey @Afamefuna_Anigbo!

Not yet, but I built something today that has to be tested on other devices first. I hope to release it tomorrow :slight_smile:


#3

Just published the feature to NPM.

Hereā€™s an example:


app.setHandler({
    async LAUNCH() {

        const availableSurfaces = this.$googleAction.getAvailableSurfaces();

        if (availableSurfaces.includes('actions.capability.SCREEN_OUTPUT')) {
            this.$googleAction.newSurface(
                ['actions.capability.SCREEN_OUTPUT'],
                'Let\'s move you to a screen device for cards and other visual responses',
                'Title')
        }
    },

    ON_NEW_SURFACE() {

        if (this.$googleAction.isNewSurfaceConfirmed()) {
            this.tell('Hello Smartphone');
        } else {
            this.tell('Hello Speaker');
        }
    }

});

#4

Thanks @AlexSwe!

Will try this out and revert!

Regards,
Afam


#5

Thanks once more, @AlexSwe. Iā€™m using this approach to get users back into the account linking flow. On Google, there isnā€™t a way to show an account linking card once the action is linked. Iā€™m transferring them from smart speaker to phone if necessary, then showing a link out suggestion to a web page with a redirect to the Google Actionā€™s page and telling the user to press the ā€œunlinkā€ button and then the ā€œlinkā€ button. Very complicated comparing to simply showing a link card in the Alexa app and telling the user to open the app.


#6

Is there some sort of equivalent here for Alexa ?


#7

For Alexa, you can simply show a LinkAccount card: https://developer.amazon.com/en-US/docs/alexa/custom-skills/include-a-card-in-your-skills-response.html#define-a-card-for-use-with-account-linking


#8

If youā€™re asking about transferring a conversation from an Echo device to the Alexa app on a phone, Iā€™m pretty sure thatā€™s not possible.


#9

Did something break here with any recent updates, I canā€™t seem to get this to work since. @AlexSwe Maybe something with the recent legacy shift


#10

Google released the new Action Builder. But it shouldnā€™t break the old functionality.

Can you provide more details? How did you implement it (relevant code parts). What are the request/response JSONs?


#11

@AlexSwe So it seems to work up until you need confirmation from Google, you get the last response right after the last query (What is it) but when you answer ā€œyesā€ as a confirmation we donā€™t get anything back. No request or anything and it ends the conversation.

const availableSurfaces = this.$googleAction.hasWebBrowserInterface();

            if (!availableSurfaces) {
                // this.followUpState('ANYWHERECARE_STATE.AnywhereCareVideo')
                this.$googleAction.newSurface(
                    ['actions.capability.SCREEN_OUTPUT'],
                    this.t('WhatIsIt'),
                    'Using UPMC Anywhere Care Virtual Visits')
            } 

}

AnywhereCareVideo: {
        YesIntent() {
            let title = 'Using UPMC Anywhere Care Virtual Visits';
            let content = 'https://www.youtube.com/watch?v=QvF23V0AkRU';
            if(this.isGoogleAction()){
                this.$googleAction.showLinkOutSuggestion(title, content);
                this.$googleAction.displayText(title).tell('Great! Click on the link below!')
            } else {
                this.$alexaSkill.showSimpleCard(title, content);
            }
        },

         NoIntent() {
            this.ask(this.t('UniversalNo'));
         },
    },

Maybe related to this ? https://github.com/actions-on-google/actions-on-google-nodejs/issues/401


#12

Did you enable the webhook in the Default Fallback Intent settings?


#13

@AlexSwe Yep it was already on and still didnā€™t work :frowning: this only seems to have broken very recently. So this is what I meant, thereā€™s no request coming back from the Google side.


#14

Can you try it again? It didnā€™t work in my sample project yesterday, but it is working now.

And I may see the problem here: You need to add something to your handler. Hereā€™s an example

app.setHandler({
	async LAUNCH() {
		const availableSurfaces = this.$googleAction.getAvailableSurfaces();
		if (availableSurfaces.includes('actions.capability.SCREEN_OUTPUT')) {
			this.$googleAction.newSurface(
				['actions.capability.SCREEN_OUTPUT'],
				"Let's move you to a screen device for cards and other visual responses",
				'Title'
			);
		}
	},

	ON_NEW_SURFACE() {
		if (this.$googleAction.isNewSurfaceConfirmed()) {
			this.tell('Hello Smartphone');
		} else {
			this.tell('Hello Speaker');
		}
	},
});


#15

Weird yea it seems to be working for me now too. I wonder what happened.


#16

@AlexSwe Is there a possibility of having the notification sent but the user remain on the initial device and then see the link on their phone later ?


#17

Just tested, I donā€™t think itā€™s possible. Whatā€™s the use case? You could take a look at push notifications (https://www.jovo.tech/tutorials/google-action-notifications)


#18

Hey @AlexSwe,

Do you know how I could make this work with the actions builder/jovo-platform-googleassistantconv? this.$googleAction.getAvailableSurfaces() returns undefined (got around this using hasWebBrowserInterface), and I get a ā€˜not a functionā€™ error for this.$googleAction.newSurface.


#19

Hey @Rik

Looks like this.$googleAction.getAvailableSurfaces() is broken. What do you need this.$googleAction.newSurface for?

You can access all capabilities via this.$googleAction.$request.device.capabilities


#20

We wanted to move the user/session to a mobile phone so we could send them notifications.