Multi surface conversations

google-assistant

#1

Hi guys,

Is it possible to have multi surface conversations on the google device as explained in the link below?

https://developers.google.com/actions/assistant/surface-capabilities

I want to be able to start a conversation on google home, and continue on my phone. Is this possible?

Regards,
Afam


#2

Hey @Afamefuna_Anigbo!

Not yet, but I built something today that has to be tested on other devices first. I hope to release it tomorrow :slight_smile:


#3

Just published the feature to NPM.

Here’s an example:


app.setHandler({
    async LAUNCH() {

        const availableSurfaces = this.$googleAction.getAvailableSurfaces();

        if (availableSurfaces.includes('actions.capability.SCREEN_OUTPUT')) {
            this.$googleAction.newSurface(
                ['actions.capability.SCREEN_OUTPUT'],
                'Let\'s move you to a screen device for cards and other visual responses',
                'Title')
        }
    },

    ON_NEW_SURFACE() {

        if (this.$googleAction.isNewSurfaceConfirmed()) {
            this.tell('Hello Smartphone');
        } else {
            this.tell('Hello Speaker');
        }
    }

});

#4

Thanks @AlexSwe!

Will try this out and revert!

Regards,
Afam


#5

Thanks once more, @AlexSwe. I’m using this approach to get users back into the account linking flow. On Google, there isn’t a way to show an account linking card once the action is linked. I’m transferring them from smart speaker to phone if necessary, then showing a link out suggestion to a web page with a redirect to the Google Action’s page and telling the user to press the “unlink” button and then the “link” button. Very complicated comparing to simply showing a link card in the Alexa app and telling the user to open the app.


#6

Is there some sort of equivalent here for Alexa ?


#7

For Alexa, you can simply show a LinkAccount card: https://developer.amazon.com/en-US/docs/alexa/custom-skills/include-a-card-in-your-skills-response.html#define-a-card-for-use-with-account-linking


#8

If you’re asking about transferring a conversation from an Echo device to the Alexa app on a phone, I’m pretty sure that’s not possible.


#9

Did something break here with any recent updates, I can’t seem to get this to work since. @AlexSwe Maybe something with the recent legacy shift


#10

Google released the new Action Builder. But it shouldn’t break the old functionality.

Can you provide more details? How did you implement it (relevant code parts). What are the request/response JSONs?


#11

@AlexSwe So it seems to work up until you need confirmation from Google, you get the last response right after the last query (What is it) but when you answer “yes” as a confirmation we don’t get anything back. No request or anything and it ends the conversation.

const availableSurfaces = this.$googleAction.hasWebBrowserInterface();

            if (!availableSurfaces) {
                // this.followUpState('ANYWHERECARE_STATE.AnywhereCareVideo')
                this.$googleAction.newSurface(
                    ['actions.capability.SCREEN_OUTPUT'],
                    this.t('WhatIsIt'),
                    'Using UPMC Anywhere Care Virtual Visits')
            } 

}

AnywhereCareVideo: {
        YesIntent() {
            let title = 'Using UPMC Anywhere Care Virtual Visits';
            let content = 'https://www.youtube.com/watch?v=QvF23V0AkRU';
            if(this.isGoogleAction()){
                this.$googleAction.showLinkOutSuggestion(title, content);
                this.$googleAction.displayText(title).tell('Great! Click on the link below!')
            } else {
                this.$alexaSkill.showSimpleCard(title, content);
            }
        },

         NoIntent() {
            this.ask(this.t('UniversalNo'));
         },
    },

Maybe related to this ? https://github.com/actions-on-google/actions-on-google-nodejs/issues/401


#12

Did you enable the webhook in the Default Fallback Intent settings?


#13

@AlexSwe Yep it was already on and still didn’t work :frowning: this only seems to have broken very recently. So this is what I meant, there’s no request coming back from the Google side.


#14

Can you try it again? It didn’t work in my sample project yesterday, but it is working now.

And I may see the problem here: You need to add something to your handler. Here’s an example

app.setHandler({
	async LAUNCH() {
		const availableSurfaces = this.$googleAction.getAvailableSurfaces();
		if (availableSurfaces.includes('actions.capability.SCREEN_OUTPUT')) {
			this.$googleAction.newSurface(
				['actions.capability.SCREEN_OUTPUT'],
				"Let's move you to a screen device for cards and other visual responses",
				'Title'
			);
		}
	},

	ON_NEW_SURFACE() {
		if (this.$googleAction.isNewSurfaceConfirmed()) {
			this.tell('Hello Smartphone');
		} else {
			this.tell('Hello Speaker');
		}
	},
});


#15

Weird yea it seems to be working for me now too. I wonder what happened.


#16

@AlexSwe Is there a possibility of having the notification sent but the user remain on the initial device and then see the link on their phone later ?


#17

Just tested, I don’t think it’s possible. What’s the use case? You could take a look at push notifications (https://www.jovo.tech/tutorials/google-action-notifications)


#18

Hey @AlexSwe,

Do you know how I could make this work with the actions builder/jovo-platform-googleassistantconv? this.$googleAction.getAvailableSurfaces() returns undefined (got around this using hasWebBrowserInterface), and I get a ‘not a function’ error for this.$googleAction.newSurface.


#19

Hey @Rik

Looks like this.$googleAction.getAvailableSurfaces() is broken. What do you need this.$googleAction.newSurface for?

You can access all capabilities via this.$googleAction.$request.device.capabilities


#20

We wanted to move the user/session to a mobile phone so we could send them notifications.