Feature Proposal: Conversational Components


#1

Feature Proposal: Conversational Components

We’re back with another feature proposal. This time we want to talk about Conversational Components.

We’ve been discussing this idea internally for a while, and it has also come up in our community a few times. Here’s a neat quote by @marktucker explaining the gist of the idea: “packaged, multi-turn conversations with accompanying language model, logic, and responses. For example, for In-Skill Purchasing there is a point where the skill developer hands over the conversation to Amazon so that the user confirms a purchase and their account is charged outside the skill code.”

The idea behind these components is to provide solutions for recurring problems you might encounter while creating voice apps.

As an example let’s say at a given time you need your user’s phone number. Instead of setting up the language model, adding the logic to your handler, creating the speech output, etc. you simply install the component and delegate the task to it.


How It Works

First of all, you would have to install the component. These have to be manually installed using npm:

$ npm install <component> --save

After that, you would use a new Jovo CLI command to extract the necessary files from the package into your project, i.e. ./models, ./src/

$ jovo prepare <component>

We are not sure yet, whether we want to place the files into their respective folders inside a components sub-directory, or bundle everything up inside a components directory in the root folder of your project and have sub-directories for each component containing all of its files:

./models/components/<component>
./src/i18n/components/<component>
./src/components/<component>

// OR

./components/<component>/models
./components/<component>/src/i18n
./components/<component>/src

But, after that you would enable the component using the new app.useComponent(componentName) function:

// src/app.js

app.useComponent('COMPONENT_PHONE_NUMBER');

That’s it for the installation.

Now you could simply delegate the task to the component using the new delegate(componentName, onCompleteIntent) function:

// src/app.js

PhoneNumberIntent() {
    this.delegate('COMPONENT_PHONE_NUMBER', 'CompletedIntent');
},

CompletedIntent() {

}

After the component finished its task, it will route back to the intent specified in onCompleteIntent and you will be able to access the result of the process using the new $components object:

CompletedIntent() {
    this.$components.COMPONENT_PHONE_NUMBER.$response
}

This response will contain a status property informing you whether the process was SUCCESSFUL, encountered an ERROR or was REJECTED by the user, i.e. user called the StopIntent at some point. Besides that, it would contain a data object, which will have the data stored you initially needed. That object will have a different structure depending on the component.

{
    "status": "SUCCESSFUL" | "ERROR" | "REJECTED",
    "data": {
        ...
    }
}

Third-Party Components

To make it easy for third-party developers to create and share their own components, they will simply be npm packages. The entry point will specify the path to the models and src folders, which will be used by the CLI to distribute them to their respective places inside your project using the $ jovo prepare <component> command.


Problems

One of the most likely problems you will encounter when using components is the corruption of the language model. For example, you’re not allowed to use the same utterance in multiple intents, or you might already have a YesIntent setup, but the component has its own. In these cases, the CLI would issue a warning, which will help you manually fix the issue.


Google Assistant list selection on surfaces with no screen
#2

I like it so far.

Feedback:

  • I like the structure where i18n, models, and src are under components/<component>/
  • For each of the Intents used internally in the model there might be conflicts with Intents the skill already has. The component needs to allow a mapping from from its expected intents to ones from the skill
    intentMap: {
        'COMPONENT_PHONE_NUMBER_YesIntent': 'YesIntent',
    },
  • Intent names and i18n key names should be prefixed with the component name (ex: COMPONENT_PHONE_NUMBER_YesIntent, COMPONENT_PHONE_NUMBER_AskPhoneSpeechKey)

  • Would the language model json file for a component be in the same jovo format as models/en-US.json?

  • Would the language model json file for a component be automatically merged with models/en-US.json on jovo build and then saved to /platforms/?

  • Would the following be an expanded sample?

    CompletedIntent() {
        const response = this.$components.COMPONENT_PHONE_NUMBER.$response;

        if (response.status === 'SUCCESSFUL') {
            const phoneNumber = response.data.phoneNumber;
            // handle success
        } else if (response.status === 'REJECTED') {
            // handle rejected
        } else {
            // handle error
        }
    }

#3

Thanks a lot for your thoughts, @marktucker :muscle:

Yes, this makes sense. Today we talked about that each Jovo Component would look similar to a mini-Jovo-app with a models folder and a src folder that includes the logic (in e.g. an index.js + additional files/modules) and a config.js. I could definitely see an intentmap as an important part of this

Makes sense

Yes

Yes

  • Would the following be an expanded sample?
    CompletedIntent() {
        const response = this.$components.COMPONENT_PHONE_NUMBER.$response;

        if (response.status === 'SUCCESSFUL') {
            const phoneNumber = response.data.phoneNumber;
            // handle success
        } else if (response.status === 'REJECTED') {
            // handle rejected
        } else {
            // handle error
        }
    }

Yes!


#4

When you call $ jovo prepare <component> will that act as a ‘code generator’ so that under `component/src’ will be the code for each handler and state?

Or will the component be treated as a blackbox?


#5

We don’t want to treat it as a black box because we think that there are many things that people might want to customize. So yes, the current idea was to add it to the src folder.

The goal is to have enough configurations though so that people don’t have to dive into the handler code to make updates.


#6

I see that. So there will be a substantial config.js file with buttons and knobs to customize?

There needs to be a way to customize without updating the .js files under src for the component. Otherwise updating the component via npm from v1 to v2 will wipe out developer modifications.


#7

Should also allow for developers to organize their code into components even if they don’t want to publish on npm. Corporate-wide components and even project-level components.


pinned globally #8

#9

Yes, I believe this would be necessary, at least for larger components.

Agreed!

Oh yes, definitely. We’ve seen this internally that we do a lot of copy-paste between projects we’re working on


#10

I love this idea! It will be ready next week sometime yes? :stuck_out_tongue:


#11

Haha, great to hear that @natrixx!

What types of components would you like to see?

Here are a few ideas:


#12

Those all sound like great ideas! I would likely even use the collect zip one.

I currently have a google sign in workflow setup for my google action, and I think that would make a good component.

I also wonder about something for dialog delegation intents, but I have barely scratched the surface of that topic on a personal project, so perhaps its not super viable.


#13

I’m thinking about this as well, if a component could be created for slot filling. This would require more configuration from the users though, maybe even at the moment of delegating to the component, not on a config.js level.

Would it make sense to have the delegate method call look like this?

this.delegate('COMPONENT_NAME', options);

And the options would then differ for each component? Maybe some components don’t need an ON_COMPLETE_INTENT, others might need more data to work with (which slots to fill, for example).

It could then look like this for the phone number:

this.delegate('COMPONENT_PHONE_NUMBER', { onComplete: 'CompletedIntent' });

#14

yea, that seems like it would work to me!


#15

I’d love to see a Google Calendar Integrated Scheduling component

We’ve barely scratched the surface of the topic, but we’re designing a skill for our own needs, we are meeting with local businesses and schedule via email frequently. Voicefirst would use this component and then also set up our customers with the same capabilities.

“If you’d like to schedule a meeting with us, just ask Alexa.”


#16

That’s a great idea @rjolayolay :+1:

I could see more components like this:

  • Schedule a meeting
  • Be connected to/get a call from customer support
  • Report a bug

#17

By the way, the concept of something like “conversational modules” was also discussed in this great blog post by Jeff Smith almost 1.5 years ago:


#18

Yeah, I like this idea a lot. Kind of the voice equivalent of React UI components. I think this would move us closer to having “patterns” that become the expected/accepted way that common VUI/VUX things are done. So, it’s not just better for developers - it’s better for users too.


#19

I was just talking to @jan about a conversational component that acts as a list selector where a user can select from several choices. This is already implemented in Google Actions as a list selector, where users can select an element by saying its name or Ordinal value. You can then access the selected element in the ON_ELEMENT_SELECTED() handler.

However this only words on Google Assistant devices that have a screen, so no Google Home. You can have a component take a list of choices and return the users choice, letting them select an it by saying its name, or by ordinal value. This would be useful if the user needs to select a song, article, or podcast title.

This is essentially already implemented on Google Assistant, except it does not work on surfaces with no screen. I’d love to hear if someone already knows how to implement a list selector on surfaces with no screen.


#20

Had a look at the pull request for CC (https://github.com/jovotech/jovo-framework/commit/215fe5e8f3e56203abbc29ae27195747433a2c7a) and thought of a few things.

Good work @Kaan_Kilic!

Thoughts and Questions:

  1. Is upppercase (PHONE_NUMBER) the convention or could I as a developer create a component called PhoneNumberUSComponent?
  2. The delegate/completed handler pair makes sense: this.delegate('PHONE_NUMBER', 'CompletedIntent'); paired with CompletedIntent() {}
  3. Would the pair work inside a state? Do you foresee a situation where the delegate can be in one state (or no state) and the completed handler in another state?
  4. Is the component name used by the class, in the useComponents statement, in the call to delegate, off of this.$components and elsewhere just convention? How is it enforced?
  5. In app’s config.js, can other component values be set and used throughout the component? For example, the count of fails before the component switches to sequence mode?
  6. Does the config for the component get merged into the config for the app? In this example, the IntentMaps?
  7. Will the component configs honor stages? config.js, config.local.js, config.dev.js, config.test.js, config.prod.js?
  8. Should the namespacing of the component translations follow a format similar to ‘org-component-name’ (ex: jovo-component-phonenumber or JovoComponentPhoneNumber or MyCorpComponentPhoneNumber or MyCorpPhoneNumber)
  9. Will the component model also follow the main Jovo model (not platform) with the ability to be named en.json and then have it properly merge out into en-US, en-GB, etc. based on the setting in project.js?
  alexaSkill: {
    nlu: {
      name: 'alexa',
      lang: {
        en: ['en-US', 'en-GB', 'en-IN', 'en-CA', 'en-AU'],
      },
    },
  1. Why does the component module use AMAZON.Number for the COMPONENT_PHONE_NUMBER_PhoneNumberIntent instead of AMAZON.PhoneNumber? (Oh, it is US-only)
  2. How will differences in available built-in intents or slots be handled in a component that is to be used in multiple locations?
  3. Having every component have a START handler as its entry point make sense.
  4. Should every intent for a component be in a STATE? For example, the root of the PHONE_NUMBER component has Yes and No intents. Using this component in a skill that also has Yes and No intents would be problematic. Maybe the START handler is outside of a STATE but everything else is in one?
  5. I currently have an implementation that has 2 different intents for numbers. One uses AMAZON.Number slot type and the other AMAZON.PhoneNumber. Because of Alexa’s lack of Intent Context, I have to include both intents in the state when asking for a phone number because sometimes one is chosen over the other. This same thing can happen with AMAZON.Time. All will need to be included that are also included in the main component because the language model is one.
  6. Any STATE in a component should have an Unhandled and FallbackIntent (maps to AMAZON.FallbackIntent). Often a StopCancelIntent (maps to AMAZON.StopIntent and AMAZON.CancelIntent) is also useful.
  7. When I have handled phone number in the past, I have checked if it is valid and sent the user back to re-enter before asking to confirm: Ask, Validate, Confirm
  8. I also handle the US situation when a user gives a 7-digit phone number instead of 10, it is assumed they forgot to say the area code. They are then prompted for the area code. This will be different in different countries.
  9. I could see US_PHONE_NUMBER and INTL_PHONE_NUMBER components
  10. Is the component outside the main src tree because the idea is that component would be separate npm modules?
  11. Can we use removeState, followUpState, toStateIntent, and toStatelessIntent in a component?

There will likely be more feedback going forward.

I really like what you are doing here and am being so detailed because I care about the framework and this feature.

@Kaan_Kilic you have done a great service getting this initial version out.

Thanks!

Mark