followUpState not working at Google Assistant

google-assistant
dialogflow

#1

Hi guys!

Yesterday something really strange started to happen in some Google Actions.

Simply the state set by “followUpState” is not working as usual.
For example:

  1. Do you want some help? (followUpState(‘somecontext’).ask…
  2. YesIntent triggered is the “global” one and not the “somecontext” one.

It´s been happening since yesterday. Take a look at the playloads:

ASSISTANT: 2 answer option: Sim ou Não (yes or no)

{
        "fulfillmentText": "<audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Fas_36-op1.mp3\">Sabia que os dentistas recomendam mudar de escova a cada três meses? Ah! Também é muito importante fazer a troca após um resfriado, gripe, infecção na boca ou dor de garganta. Isso porque os germes podem se alojar nas cerdas da escova e levar à reinfecção. Posso lembrar quando você deve trocar a sua escova?</audio>",
        "outputContexts": [
                {
                        "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/__system_counters__",
                        "lifespanCount": 1,
                        "parameters": {
                                "no-input": 0,
                                "no-match": 1
                        }
                },
                {
                        "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/_jovo_session_xeplc",
                        "lifespanCount": 1,
                        "parameters": {
                                "_JOVO_STATE_": "Agenda_PossoLembrar"
                        }
                }
        ],
        "payload": {
                "google": {
                        "expectUserResponse": true,
                        "richResponse": {
                                "items": [
                                        {
                                                "simpleResponse": {
                                                        "ssml": "<speak><audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Fas_36-op1.mp3\">Sabia que os dentistas recomendam mudar de escova a cada três meses? Ah! Também é muito importante fazer a troca após um resfriado, gripe, infecção na boca ou dor de garganta. Isso porque os germes podem se alojar nas cerdas da escova e levar à reinfecção. Posso lembrar quando você deve trocar a sua escova?</audio></speak>"
                                                }
                                        }
                                ],
                                "suggestions": [
                                        {
                                                "title": "Sim"
                                        },
                                        {
                                                "title": "Não"
                                        }
                                ]
                        },
                        "noInputPrompts": [
                                {
                                        "ssml": "<speak><audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Fas_36-op1.mp3\">Sabia que os dentistas recomendam mudar de escova a cada três meses? Ah! Também é muito importante fazer a troca após um resfriado, gripe, infecção na boca ou dor de garganta. Isso porque os germes podem se alojar nas cerdas da escova e levar à reinfecção. Posso lembrar quando você deve trocar a sua escova?</audio></speak>"
                                }
                        ],
                        "userStorage": "{\"userId\":\"4be8b8e4-1a50-40f2-9f84-690a213ed707\"}"
                }
        }
}

USER: Sim (yes).

{
    "responseId": "97fe6566-f46a-4905-9977-b5bb7c2a1839-d794dba9",
    "queryResult": {
            "queryText": "Sim",
            "parameters": {},
            "allRequiredParamsPresent": true,
            "outputContexts": [
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_account_linking"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_media_response_audio"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_web_browser"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_audio_output"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/actions_capability_screen_output"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/google_assistant_input_type_touch"
                    },
                    {
                            "name": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc/contexts/__system_counters__",
                            "parameters": {
                                    "no-input": 0,
                                    "no-match": 0
                            }
                    }
            ],
            "intent": {
                    "name": "projects/colgate-254520/agent/intents/6b2685ca-0e0d-4276-a1b8-b152e36df93b",
                    "displayName": "YesIntent"
            },
            "intentDetectionConfidence": 1,
            "languageCode": "pt-br"
    },
    "originalDetectIntentRequest": {
            "source": "google",
            "version": "2",
            "payload": {
                    "user": {
                            "locale": "pt-BR",
                            "lastSeen": "2020-08-20T12:05:43Z",
                            "userStorage": "{\"userId\":\"4be8b8e4-1a50-40f2-9f84-690a213ed707\"}",
                            "userVerificationStatus": "VERIFIED"
                    },
                    "conversation": {
                            "conversationId": "ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc",
                            "type": "ACTIVE",
                            "conversationToken": "[\"__system_counters__\",\"_jovo_session_xeplc\"]"
                    },
                    "inputs": [
                            {
                                    "intent": "actions.intent.TEXT",
                                    "rawInputs": [
                                            {
                                                    "inputType": "TOUCH",
                                                    "query": "Sim"
                                            }
                                    ],
                                    "arguments": [
                                            {
                                                    "name": "text",
                                                    "rawText": "Sim",
                                                    "textValue": "Sim"
                                            }
                                    ]
                            }
                    ],
                    "surface": {
                            "capabilities": [
                                    {
                                            "name": "actions.capability.ACCOUNT_LINKING"
                                    },
                                    {
                                            "name": "actions.capability.MEDIA_RESPONSE_AUDIO"
                                    },
                                    {
                                            "name": "actions.capability.WEB_BROWSER"
                                    },
                                    {
                                            "name": "actions.capability.AUDIO_OUTPUT"
                                    },
                                    {
                                            "name": "actions.capability.SCREEN_OUTPUT"
                                    }
                            ]
                    },
                    "isInSandbox": true,
                    "availableSurfaces": [
                            {
                                    "capabilities": [
                                            {
                                                    "name": "actions.capability.AUDIO_OUTPUT"
                                            },
                                            {
                                                    "name": "actions.capability.SCREEN_OUTPUT"
                                            },
                                            {
                                                    "name": "actions.capability.WEB_BROWSER"
                                            }
                                    ]
                            }
                    ],
                    "requestType": "SIMULATOR"
            }
    },
    "session": "projects/colgate-254520/agent/sessions/ABwppHFr_qbdewbPctSi8VtDgCBR8bkSQ0TEX-oIJ22erQv2F6k51Gee_afpuaGZtQoWfNNprQhvoA0Jorc"

}

ASSISTANT: The state has gone!

{
    "fulfillmentText": "<audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Aber_02-op2.mp3\">Desculpe, mas no momento só falo sobre higiene bucal básica, problemas mais comuns, estética e saúde bucal em diferentes fases da vida. Ah! E posso também agendar a troca da sua escova. O que você escolhe?</audio>",
    "outputContexts": [],
    "payload": {
            "google": {
                    "expectUserResponse": true,
                    "richResponse": {
                            "items": [
                                    {
                                            "simpleResponse": {
                                                    "ssml": "<speak><audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Aber_02-op2.mp3\">Desculpe, mas no momento só falo sobre higiene bucal básica, problemas mais comuns, estética e saúde bucal em diferentes fases da vida. Ah! E posso também agendar a troca da sua escova. O que você escolhe?</audio></speak>"
                                            }
                                    }
                            ]
                    },
                    "noInputPrompts": [
                            {
                                    "ssml": "<speak><audio src=\"https://storage.googleapis.com/colgate-files/audios/googleAction/Aber_02-op2.mp3\">Desculpe, mas no momento só falo sobre higiene bucal básica, problemas mais comuns, estética e saúde bucal em diferentes fases da vida. Ah! E posso também agendar a troca da sua escova. O que você escolhe?</audio></speak>"
                            }
                    ],
                    "systemIntent": {
                            "intent": "actions.intent.OPTION",
                            "data": {
                                    "@type": "type.googleapis.com/google.actions.v2.OptionValueSpec",
                                    "carouselSelect": {
                                            "items": [
                                                    {
                                                            "optionInfo": {
                                                                    "key": "HigieneBasica",
                                                                    "synonyms": []
                                                            },
                                                            "title": "Higiene básica",
                                                            "description": "Escovação, fio dental, flúor e Visita ao Dentista",
                                                            "image": {
                                                                    "url": "https://storage.googleapis.com/colgate-files/imagens-colgate-oralcare/cards/1_HIGIENE_BASICA.jpg",
                                                                    "accessibilityText": "Higiene básica",
                                                                    "width": 1920,
                                                                    "height": 1080
                                                            }
                                                    },
                                            ]
                                    }
                            }
                    },
                    "userStorage": "{\"userId\":\"4be8b8e4-1a50-40f2-9f84-690a213ed707\"}"
            }
    }

}

Thank you!


Google Assitant/Dialogflow is not returning to the Jovo State
#2

Same problem here: Our live action (which did not update jovo at all) is loosing the state output context from one response to the next request. Something changed on the google action side…


#3

Yes Andre! I´ve sent a support ticket to Dialogflow team as well.

It´s simply unstable, sometimes the session are kept sometimes not… :frowning:


#4

Google has already been notified and has started working to fix:
https://issuetracker.google.com/u/1/issues/165676621

Let´s wait! :neutral_face:


#5

I have good and bad news :smiley:

Good News: We have a quick workaround for that. With this configuration the session data will be stored in the database. States will work because they do not use the outputContexts in Dialogflow.

config.js

...
user: {
		sessionData: {
			enabled: true,
			data: true,
		},
}
...

Bad News: There was a bug in the implementation and It doesn’t work with Dialogflow right now.

I’m working on a fix.


#6

The fix for the workaround is published.

// config.js
...
user: {
		sessionData: {
			enabled: true,
			data: true,
		},
}
...

This will save all session data to the data base.


#7

Thanks @AlexSwe!
Saved me from a painful headache… :clap:


#8

@AlexSwe Storing session data in the database is also a good approach when including Bixby in cross platform projects as Bixby requires all session variables to be explicitly modeled.


#9

@AlexSwe Thanks. Fix tested and working. :+1:


#10

@AlexSwe fix is not working for me. Here is my fileDb after setting session data in a handler:`

{
	"userId": "7737abd6-6a6b-4bef-9259-24e21e394edc",
	"userData": {
		"data": {
			"visits": 1
		},
		"session": {
			"$data": {
				"recipes": [
					{
						"title": "Super Easy Chicken and Dumplings",
						"id": "218562",
						"cookTime": "15 Mins ",
						"ratingCount": 176,
						"serves": 8,
						"image": "https://images.media-allrecipes.com/userphotos/3438336.jpg"
					}, ......

Then the next request when I try to read that data:

[
{
“userId”: “7737abd6-6a6b-4bef-9259-24e21e394edc”,
“userData”: {
“data”: {
“visits”: 1
},
“session”: {
“$data”: {},
“lastUpdatedAt”: “2020-08-20T18:18:49.627Z”
}
}
}
]

`


#11

They fixed it. https://issuetracker.google.com/u/1/issues/165676621


#12

Hello
I have the same problem, i think the problem is not solved in DialogFlow.
Do you see an error below please ?

DiagnosisState : {

  ComplexDiagnosisIntent(){
        const responses = this.$session.$data.responses;
        if(typeof responses === 'undefined' || !responses){
            return this.toIntent('ErrorOccuredIntent');
        }
        
        
        if(responses.answer.question){
            let question = responses.answer.question;
            this.$session.$data.complex_step = 1;
            this.$session.$data.question = question;
            this.$session.$data.tree = responses.answer;
            if(this.$session.$data.lst_data.lst_items.length > 1 ){
                this.$googleAction.showSuggestionChips([this.t('YES'), this.t('NO'), this.t('SOLUTION_SEARCH.PREV')]);
                this.followUpState('ChoiceDiagnosisState').ask(`${question} ${this.t('SOLUTION_SEARCH.ASK_SOLUTION_SOLVED')} ${this.t('SOLUTION_SEARCH.BACK_LIST')} `); 
            } else {
                this.$googleAction.showSuggestionChips([this.t('YES'), this.t('NO')]);
                this.followUpState('ChoiceDiagnosisState').ask(`${question} ${this.t('SOLUTION_SEARCH.ASK_SOLUTION_SOLVED')}`); 
            }
        } else if(responses.answer.split_question){
            const step = 0;
            this.$session.$data.step = step;
            return this.toStateIntent('SplitDiagnosisState', 'SplitComplexDiagnosisIntent');
        } else {
            return this.toIntent('ErrorOccuredIntent');
        }
    },
},

ChoiceDiagnosisState:{

    YesIntent(){
        
        const responses = this.$session.$data.responses;
        const tree = this.$session.$data.tree;

        if(typeof responses === 'undefined' || !responses || typeof tree === 'undefined' || !tree){
            return this.toIntent('ErrorOccuredIntent');
        }
       
        if(tree.isYes.id){
            let flatted = [];
            let complex_step = this.$session.$data.complex_step;
            let tree_obj = flatObject(flatted,responses.answer,'isYes',complex_step)[0];
            this.$session.$data.complex_step = complex_step + 1;
            this.$session.$data.tree = tree_obj;
        } else {
            this.$session.$data.tree = tree.isYes;
        }

        return this.toStateIntent('ChoiceDiagnosisState','QuestionIntent');

    },

    NoIntent(){
        const responses = this.$session.$data.responses;
        const tree = this.$session.$data.tree;

        if(typeof responses === 'undefined' || !responses || typeof tree === 'undefined' || !tree){
            return this.toIntent('ErrorOccuredIntent');
        }
       
        if(tree.isNo.id){
            let flatted = [];
            let complex_step = this.$session.$data.complex_step;
            let tree_obj = flatObject(flatted,responses.answer,'isNo',complex_step)[0];
            this.$session.$data.complex_step = complex_step + 1;
            this.$session.$data.tree = tree_obj;
        } else {
            this.$session.$data.tree = tree.isNo;
        }

        return this.toStateIntent('ChoiceDiagnosisState','QuestionIntent');

    },

    PreviousIntent(){
        const responses = this.$session.$data.responses;
        const tree = this.$session.$data.tree;
        console.log('prev ',tree);
        if(typeof responses === 'undefined' || !responses || typeof tree === 'undefined' || !tree){
            return this.toIntent('ErrorOccuredIntent');
        }
       
        if(tree.id == 1 && this.$session.$data.lst_data.lst_items.length > 1 ){
            return this.toStateIntent('DiagnosisState.ListState','ListChoiceDiagnosisIntent');
        } else{
            let flatted = [];
            let complex_step = this.$session.$data.complex_step;
            let tree_obj = flatObject(flatted,responses.answer,'isNo',complex_step)[0];
            this.$session.$data.complex_step = complex_step + 1;
            this.$session.$data.tree = tree_obj;
        }
        
    },

    QuestionIntent(){
        const tree = this.$session.$data.tree;
        

        if(tree.action){
            let branch = tree.action;
            switch(branch.type){
                case 'OrderPart':
                    this.$session.$data.part = branch.part;
                    this.$googleAction.showSuggestionChips([this.t('YES'),this.t('NO')]);
                    this.followUpState('ComplexOrderState').ask(this.t('ORDER_PART_FLOW.ASK_ORDER'));
                    break;
                case 'CreateTicket':
                    return this.toStateIntent('TicketIntentState', 'CreateTicketIntent');
                case 'Section':
                    this.$session.$data.message = branch.message;
                    this.$googleAction.showSuggestionChips([this.t('YES'),this.t('NO')]);
                    this.followUpState('ComplexSectionState').ask(this.t('SOLUTION_SEARCH.ASK_SECTION'));
                    break;
                case 'ProblemSolved':
                    return this.toStateIntent('ProblemSolvedState', 'ProblemSolvedServiceIntent');
                default:
                    return this.toIntent('ErrorOccuredIntent');
            }
        } else if(tree.question){
            let question = tree.question;
            this.$session.$data.question = question;
            this.$googleAction.showSuggestionChips([this.t('YES'),this.t('NO'), this.t('SOLUTION_SEARCH.PREV')]);
            this.followUpState('ChoiceDiagnosisState').ask(`${question} ${this.t('SOLUTION_SEARCH.ASK_SOLUTION_SOLVED')} ${this.t('SOLUTION_SEARCH.BACK_STEP')}`);
        } else {
            return this.toIntent('ErrorOccuredIntent');
        }
    },

    Unhandled() {
        const question = this.$session.$data.question;
        this.$googleAction.showSuggestionChips([this.t('YES'),this.t('NO')]);
        this.ask(`${question} ${this.t('SOLUTION_SEARCH.ASK_SOLUTION_SOLVED')}`);
    },
},

I try that
user: {
sessionData: {
enabled: true,
data: true,
},

and to save i use this.$session.data.key = “test” but it is not working

Thanks


#13

Hey @Romain.Rivolta
You posted a lot of code :slight_smile: Where’s the relevant part?

Could you provide the request and response JSONs?


#14

Hey @AlexSwe

Here the response

{
“responseMetadata”: {
“status”: {
“code”: 10,
“message”: “Failed to parse Dialogflow response into AppResponse because of invalid platform response: Could not find a RichResponse or SystemIntent in the platform response for agentId: eaafdcea-a032-4a88-a31e-b762308613c6 and intentId: 7a2fe49f-16fd-4917-b288-7eeb5673b777. WebhookStatus: code: 4\nmessage: “Webhook call failed. Error: DEADLINE_EXCEEDED.”\n.”
}
}
}

Request bellow

{
“user”: {
“locale”: “fr-CA”,
“lastSeen”: “2020-08-31T07:06:50Z”,
“userStorage”: “{“userId”:“87bfb5f2-55b3-4f47-8ece-5e9d6b6e4608”}”,
“userVerificationStatus”: “VERIFIED”
},
“conversation”: {
“conversationId”: “ABwppHFpoaq7FEZCJbo8G9bkFc_uOs4ZeCFEs3QKSGtEk_x9hDOR5ipMzN9cDPYnTuMPuvn9zyEwgFoyQlAo_m-8vd9wbOHL”,
“type”: “ACTIVE”,
“conversationToken”: “[”_jovo_session_njwwy"]"
},
“inputs”: [
{
“intent”: “actions.intent.TEXT”,
“rawInputs”: [
{
“inputType”: “VOICE”,
“query”: “Non”
}
],
“arguments”: [
{
“name”: “text”,
“rawText”: “Non”,
“textValue”: “Non”
}
]
}
],
“surface”: {
“capabilities”: [
{
“name”: “actions.capability.AUDIO_OUTPUT”
},
{
“name”: “actions.capability.MEDIA_RESPONSE_AUDIO”
},
{
“name”: “actions.capability.ACCOUNT_LINKING”
},
{
“name”: “actions.capability.SCREEN_OUTPUT”
}
]
},
“isInSandbox”: true,
“availableSurfaces”: [
{
“capabilities”: [
{
“name”: “actions.capability.AUDIO_OUTPUT”
},
{
“name”: “actions.capability.SCREEN_OUTPUT”
},
{
“name”: “actions.capability.WEB_BROWSER”
}
]
}
],
“requestType”: “SIMULATOR”
}


#15

Looks like another error to me. I have never seen this one.