followUpState not working at Google Assistant

rrubini · 2020-08-20 12:26:08 UTC

Hi guys!

Yesterday something really strange started to happen in some Google Actions.

Simply the state set by “followUpState” is not working as usual.
For example:

Do you want some help? (followUpState(‘somecontext’).ask…
YesIntent triggered is the “global” one and not the “somecontext” one.

It´s been happening since yesterday. Take a look at the playloads:

ASSISTANT: 2 answer option: Sim ou Não (yes or no)

{
        "fulfillmentText": "<audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Fas_36-op1.mp3\">Sabia que os dentistas recomendam mudar de escova a cada três meses? Ah! Também é muito importante fazer a troca após um resfriado, gripe, infecção na boca ou dor de garganta. Isso porque os germes podem se alojar nas cerdas da escova e levar à reinfecção. Posso lembrar quando você deve trocar a sua escova?</audio>",
        "outputContexts": [
                {
                        "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/__system_counters__",
                        "lifespanCount": 1,
                        "parameters": {
                                "no-input": 0,
                                "no-match": 1
                        }
                },
                {
                        "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/_jovo_session_xeplc",
                        "lifespanCount": 1,
                        "parameters": {
                                "_JOVO_STATE_": "Agenda_PossoLembrar"
                        }
                }
        ],
        "payload": {
                "google": {
                        "expectUserResponse": true,
                        "richResponse": {
                                "items": [
                                        {
                                                "simpleResponse": {
                                                        "ssml": "<speak><audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Fas_36-op1.mp3\">Sabia que os dentistas recomendam mudar de escova a cada três meses? Ah! Também é muito importante fazer a troca após um resfriado, gripe, infecção na boca ou dor de garganta. Isso porque os germes podem se alojar nas cerdas da escova e levar à reinfecção. Posso lembrar quando você deve trocar a sua escova?</audio></speak>"
                                                }
                                        }
                                ],
                                "suggestions": [
                                        {
                                                "title": "Sim"
                                        },
                                        {
                                                "title": "Não"
                                        }
                                ]
                        },
                        "noInputPrompts": [
                                {
                                        "ssml": "<speak><audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Fas_36-op1.mp3\">Sabia que os dentistas recomendam mudar de escova a cada três meses? Ah! Também é muito importante fazer a troca após um resfriado, gripe, infecção na boca ou dor de garganta. Isso porque os germes podem se alojar nas cerdas da escova e levar à reinfecção. Posso lembrar quando você deve trocar a sua escova?</audio></speak>"
                                }
                        ],
                        "userStorage": "{\"userId\":\"4be8b8e4-1a50-40f2-9f84-690a213ed707\"}"
                }
        }
}

USER: Sim (yes).

{
    "responseId": "97fe6566-f46a-4905-9977-b5bb7c2a1839-d794dba9",
    "queryResult": {
            "queryText": "Sim",
            "parameters": {},
            "allRequiredParamsPresent": true,
            "outputContexts": [
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_account_linking"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_media_response_audio"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_web_browser"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_audio_output"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_screen_output"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/google_assistant_input_type_touch"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/__system_counters__",
                            "parameters": {
                                    "no-input": 0,
                                    "no-match": 0
                            }
                    }
            ],
            "intent": {
                    "name": "projects/colgate-254520/agent/intents/6b2685ca-0e0d-4276-a1b8-b152e36df93b",
                    "displayName": "YesIntent"
            },
            "intentDetectionConfidence": 1,
            "languageCode": "pt-br"
    },
    "originalDetectIntentRequest": {
            "source": "google",
            "version": "2",
            "payload": {
                    "user": {
                            "locale": "pt-BR",
                            "lastSeen": "2020-08-20T12:05:43Z",
                            "userStorage": "{\"userId\":\"4be8b8e4-1a50-40f2-9f84-690a213ed707\"}",
                            "userVerificationStatus": "VERIFIED"
                    },
                    "conversation": {
                            "conversationId": "ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc",
                            "type": "ACTIVE",
                            "conversationToken": "[\"__system_counters__\",\"_jovo_session_xeplc\"]"
                    },
                    "inputs": [
                            {
                                    "intent": "actions.intent.TEXT",
                                    "rawInputs": [
                                            {
                                                    "inputType": "TOUCH",
                                                    "query": "Sim"
                                            }
                                    ],
                                    "arguments": [
                                            {
                                                    "name": "text",
                                                    "rawText": "Sim",
                                                    "textValue": "Sim"
                                            }
                                    ]
                            }
                    ],
                    "surface": {
                            "capabilities": [
                                    {
                                            "name": "actions.capability.ACCOUNT_LINKING"
                                    },
                                    {
                                            "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
                                    },
                                    {
                                            "name": "actions.capability.WEB_BROWSER"
                                    },
                                    {
                                            "name": "actions.capability.AUDIO_OUTPUT"
                                    },
                                    {
                                            "name": "actions.capability.SCREEN_OUTPUT"
                                    }
                            ]
                    },
                    "isInSandbox": true,
                    "availableSurfaces": [
                            {
                                    "capabilities": [
                                            {
                                                    "name": "actions.capability.AUDIO_OUTPUT"
                                            },
                                            {
                                                    "name": "actions.capability.SCREEN_OUTPUT"
                                            },
                                            {
                                                    "name": "actions.capability.WEB_BROWSER"
                                            }
                                    ]
                            }
                    ],
                    "requestType": "SIMULATOR"
            }
    },
    "session": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc"

}

ASSISTANT: The state has gone!

{
    "fulfillmentText": "<audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Aber_02-op2.mp3\">Desculpe, mas no momento só falo sobre higiene bucal básica, problemas mais comuns, estética e saúde bucal em diferentes fases da vida. Ah! E posso também agendar a troca da sua escova. O que você escolhe?</audio>",
    "outputContexts": [],
    "payload": {
            "google": {
                    "expectUserResponse": true,
                    "richResponse": {
                            "items": [
                                    {
                                            "simpleResponse": {
                                                    "ssml": "<speak><audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Aber_02-op2.mp3\">Desculpe, mas no momento só falo sobre higiene bucal básica, problemas mais comuns, estética e saúde bucal em diferentes fases da vida. Ah! E posso também agendar a troca da sua escova. O que você escolhe?</audio></speak>"
                                            }
                                    }
                            ]
                    },
                    "noInputPrompts": [
                            {
                                    "ssml": "<speak><audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Aber_02-op2.mp3\">Desculpe, mas no momento só falo sobre higiene bucal básica, problemas mais comuns, estética e saúde bucal em diferentes fases da vida. Ah! E posso também agendar a troca da sua escova. O que você escolhe?</audio></speak>"
                            }
                    ],
                    "systemIntent": {
                            "intent": "actions.intent.OPTION",
                            "data": {
                                    "@type": "type.googleapis.com/google.actions.v2.OptionValueSpec",
                                    "carouselSelect": {
                                            "items": [
                                                    {
                                                            "optionInfo": {
                                                                    "key": "HigieneBasica",
                                                                    "synonyms": []
                                                            },
                                                            "title": "Higiene básica",
                                                            "description": "Escovação, fio dental, flúor e Visita ao Dentista",
                                                            "image": {
                                                                    "url": "https://storage.googleapis.com/colgate-files/imagens-colgate-oralcare/cards/1_HIGIENE_BASICA.jpg",
                                                                    "accessibilityText": "Higiene básica",
                                                                    "width": 1920,
                                                                    "height": 1080
                                                            }
                                                    },
                                            ]
                                    }
                            }
                    },
                    "userStorage": "{\"userId\":\"4be8b8e4-1a50-40f2-9f84-690a213ed707\"}"
            }
    }

}

Thank you!

Andre · 2020-08-20 12:38:40 UTC

Same problem here: Our live action (which did not update jovo at all) is loosing the state output context from one response to the next request. Something changed on the google action side…

rrubini · 2020-08-20 13:02:53 UTC

Yes Andre! I´ve sent a support ticket to Dialogflow team as well.

It´s simply unstable, sometimes the session are kept sometimes not…

rrubini · 2020-08-20 13:20:01 UTC

Google has already been notified and has started working to fix:
https://issuetracker.google.com/u/1/issues/165676621

Let´s wait!

AlexSwe · 2020-08-20 14:13:23 UTC

I have good and bad news

Good News: We have a quick workaround for that. With this configuration the session data will be stored in the database. States will work because they do not use the outputContexts in Dialogflow.

config.js

...
user: {
		sessionData: {
			enabled: true,
			data: true,
		},
}
...

Bad News: There was a bug in the implementation and It doesn’t work with Dialogflow right now.

I’m working on a fix.

AlexSwe · 2020-08-20 17:37:37 UTC

The fix for the workaround is published.

// config.js
...
user: {
		sessionData: {
			enabled: true,
			data: true,
		},
}
...

This will save all session data to the data base.

rrubini · 2020-08-20 18:03:34 UTC

Thanks @AlexSwe!
Saved me from a painful headache…

marktucker · 2020-08-20 18:13:19 UTC

@AlexSwe Storing session data in the database is also a good approach when including Bixby in cross platform projects as Bixby requires all session variables to be explicitly modeled.

pai · 2020-08-20 18:17:04 UTC

@AlexSwe Thanks. Fix tested and working.

BronxBombers · 2020-08-20 18:19:12 UTC

@AlexSwe fix is not working for me. Here is my fileDb after setting session data in a handler:`

{
	"userId": "7737abd6-6a6b-4bef-9259-24e21e394edc",
	"userData": {
		"data": {
			"visits": 1
		},
		"session": {
			"$data": {
				"recipes": [
					{
						"title": "Super Easy Chicken and Dumplings",
						"id": "218562",
						"cookTime": "15 Mins ",
						"ratingCount": 176,
						"serves": 8,
						"image": "https://images.media-allrecipes.com/userphotos/3438336.jpg"
					}, ......

Then the next request when I try to read that data:

[
{
“userId”: “7737abd6-6a6b-4bef-9259-24e21e394edc”,
“userData”: {
“data”: {
“visits”: 1
},
“session”: {
“$data”: {},
“lastUpdatedAt”: “2020-08-20T18:18:49.627Z”
}
}
}
]

`

AlexSwe · 2020-08-21 05:42:46 UTC

They fixed it. https://issuetracker.google.com/u/1/issues/165676621

Romain.Rivolta · 2020-08-28 13:42:39 UTC

Hello
I have the same problem, i think the problem is not solved in DialogFlow.
Do you see an error below please ?

DiagnosisState : {

  ComplexDiagnosisIntent(){
        const responses = this.$session.$data.responses;
        if(typeof responses === 'undefined' || !responses){
            return this.toIntent('ErrorOccuredIntent');
        }
        
        
        if(responses.answer.question){
            let question = responses.answer.question;
            this.$session.$data.complex_step = 1;
            this.$session.$data.question = question;
            this.$session.$data.tree = responses.answer;
            if(this.$session.$data.lst_data.lst_items.length > 1 ){
                this.$googleAction.showSuggestionChips([this.t('YES'), this.t('NO'), this.t('SOLUTION_SEARCH.PREV')]);
                this.followUpState('ChoiceDiagnosisState').ask(`${question} ${this.t('SOLUTION_SEARCH.ASK_SOLUTION_SOLVED')} ${this.t('SOLUTION_SEARCH.BACK_LIST')} `); 
            } else {
                this.$googleAction.showSuggestionChips([this.t('YES'), this.t('NO')]);
                this.followUpState('ChoiceDiagnosisState').ask(`${question} ${this.t('SOLUTION_SEARCH.ASK_SOLUTION_SOLVED')}`); 
            }
        } else if(responses.answer.split_question){
            const step = 0;
            this.$session.$data.step = step;
            return this.toStateIntent('SplitDiagnosisState', 'SplitComplexDiagnosisIntent');
        } else {
            return this.toIntent('ErrorOccuredIntent');
        }
    },
},

ChoiceDiagnosisState:{

    YesIntent(){
        
        const responses = this.$session.$data.responses;
        const tree = this.$session.$data.tree;

        if(typeof responses === 'undefined' || !responses || typeof tree === 'undefined' || !tree){
            return this.toIntent('ErrorOccuredIntent');
        }
       
        if(tree.isYes.id){
            let flatted = [];
            let complex_step = this.$session.$data.complex_step;
            let tree_obj = flatObject(flatted,responses.answer,'isYes',complex_step)[0];
            this.$session.$data.complex_step = complex_step + 1;
            this.$session.$data.tree = tree_obj;
        } else {
            this.$session.$data.tree = tree.isYes;
        }

        return this.toStateIntent('ChoiceDiagnosisState','QuestionIntent');

    },

    NoIntent(){
        const responses = this.$session.$data.responses;
        const tree = this.$session.$data.tree;

        if(typeof responses === 'undefined' || !responses || typeof tree === 'undefined' || !tree){
            return this.toIntent('ErrorOccuredIntent');
        }
       
        if(tree.isNo.id){
            let flatted = [];
            let complex_step = this.$session.$data.complex_step;
            let tree_obj = flatObject(flatted,responses.answer,'isNo',complex_step)[0];
            this.$session.$data.complex_step = complex_step + 1;
            this.$session.$data.tree = tree_obj;
        } else {
            this.$session.$data.tree = tree.isNo;
        }

        return this.toStateIntent('ChoiceDiagnosisState','QuestionIntent');

    },

    PreviousIntent(){
        const responses = this.$session.$data.responses;
        const tree = this.$session.$data.tree;
        console.log('prev ',tree);
        if(typeof responses === 'undefined' || !responses || typeof tree === 'undefined' || !tree){
            return this.toIntent('ErrorOccuredIntent');
        }
       
        if(tree.id == 1 && this.$session.$data.lst_data.lst_items.length > 1 ){
            return this.toStateIntent('DiagnosisState.ListState','ListChoiceDiagnosisIntent');
        } else{
            let flatted = [];
            let complex_step = this.$session.$data.complex_step;
            let tree_obj = flatObject(flatted,responses.answer,'isNo',complex_step)[0];
            this.$session.$data.complex_step = complex_step + 1;
            this.$session.$data.tree = tree_obj;
        }
        
    },

    QuestionIntent(){
        const tree = this.$session.$data.tree;
        

        if(tree.action){
            let branch = tree.action;
            switch(branch.type){
                case 'OrderPart':
                    this.$session.$data.part = branch.part;
                    this.$googleAction.showSuggestionChips([this.t('YES'),this.t('NO')]);
                    this.followUpState('ComplexOrderState').ask(this.t('ORDER_PART_FLOW.ASK_ORDER'));
                    break;
                case 'CreateTicket':
                    return this.toStateIntent('TicketIntentState', 'CreateTicketIntent');
                case 'Section':
                    this.$session.$data.message = branch.message;
                    this.$googleAction.showSuggestionChips([this.t('YES'),this.t('NO')]);
                    this.followUpState('ComplexSectionState').ask(this.t('SOLUTION_SEARCH.ASK_SECTION'));
                    break;
                case 'ProblemSolved':
                    return this.toStateIntent('ProblemSolvedState', 'ProblemSolvedServiceIntent');
                default:
                    return this.toIntent('ErrorOccuredIntent');
            }
        } else if(tree.question){
            let question = tree.question;
            this.$session.$data.question = question;
            this.$googleAction.showSuggestionChips([this.t('YES'),this.t('NO'), this.t('SOLUTION_SEARCH.PREV')]);
            this.followUpState('ChoiceDiagnosisState').ask(`${question} ${this.t('SOLUTION_SEARCH.ASK_SOLUTION_SOLVED')} ${this.t('SOLUTION_SEARCH.BACK_STEP')}`);
        } else {
            return this.toIntent('ErrorOccuredIntent');
        }
    },

    Unhandled() {
        const question = this.$session.$data.question;
        this.$googleAction.showSuggestionChips([this.t('YES'),this.t('NO')]);
        this.ask(`${question} ${this.t('SOLUTION_SEARCH.ASK_SOLUTION_SOLVED')}`);
    },
},

I try that
user: {
sessionData: {
enabled: true,
data: true,
},

and to save i use this.$session.data.key = “test” but it is not working

Thanks

AlexSwe · 2020-08-30 15:09:15 UTC

Hey @Romain.Rivolta
You posted a lot of code Where’s the relevant part?

Could you provide the request and response JSONs?

Romain.Rivolta · 2020-08-31 07:09:17 UTC

Hey @AlexSwe

Here the response

{
“responseMetadata”: {
“status”: {
“code”: 10,
“message”: “Failed to parse Dialogflow response into AppResponse because of invalid platform response: Could not find a RichResponse or SystemIntent in the platform response for agentId: eaafdcea-a032-4a88-a31e-b762308613c6 and intentId: 7a2fe49f-16fd-4917-b288-7eeb5673b777. WebhookStatus: code: 4\nmessage: “Webhook call failed. Error: DEADLINE_EXCEEDED.”\n.”
}
}
}

Request bellow

{
“user”: {
“locale”: “fr-CA”,
“lastSeen”: “2020-08-31T07:06:50Z”,
“userStorage”: “{“userId”:“87bfb5f2-55b3-4f47-8ece-5e9d6b6e4608”}”,
“userVerificationStatus”: “VERIFIED”
},
“conversation”: {
“conversationId”: “ABwppHFpoaq7FEZCJbo8G9bkFc_uOs4ZeCFEs3QKSGtEk_x9hDOR5ipMzN9cDPYnTuMPuvn9zyEwgFoyQlAo_m-8vd9wbOHL”,
“type”: “ACTIVE”,
“conversationToken”: “[”_jovo_session_njwwy"]"
},
“inputs”: [
{
“intent”: “actions.intent.TEXT”,
“rawInputs”: [
{
“inputType”: “VOICE”,
“query”: “Non”
}
],
“arguments”: [
{
“name”: “text”,
“rawText”: “Non”,
“textValue”: “Non”
}
]
}
],
“surface”: {
“capabilities”: [
{
“name”: “actions.capability.AUDIO_OUTPUT”
},
{
“name”: “actions.capability.MEDIA_RESPONSE_AUDIO”
},
{
“name”: “actions.capability.ACCOUNT_LINKING”
},
{
“name”: “actions.capability.SCREEN_OUTPUT”
}
]
},
“isInSandbox”: true,
“availableSurfaces”: [
{
“capabilities”: [
{
“name”: “actions.capability.AUDIO_OUTPUT”
},
{
“name”: “actions.capability.SCREEN_OUTPUT”
},
{
“name”: “actions.capability.WEB_BROWSER”
}
]
}
],
“requestType”: “SIMULATOR”
}

AlexSwe · 2020-08-31 07:23:02 UTC

Looks like another error to me. I have never seen this one.