Switching from local to lambda

#1

I’m trying to follow the instructions at https://www.jovo.tech/tutorials/alexa-skill-tutorial-nodejs#host-your-code-on-aws-lambda to start running my back end code on the AWS Lambda platform.

Theoretically I’ve built the zipfile and manually uploaded it to my lambda; the lambda’s code page says it’s there though at 14MB it’s loo large to edit online. I then copied the ARN from the lambda into the skill’s endpoint, confirmed that I’ve selected that rather than https, and built the model again (out of general paranoia).

However, when I make a spoken request through my device, it is still being routed to the Jovo webhook and my local Jovo session responds, as can be seen on its console.

I presume I’m missing something obvious. What else do I need to do so Alexa will invoke the lambda instead?

Obviously, if there’s additional info you need, just ask.

Possibly relevant: When I try to test the lambda manually by sending it an Alexa Start Session from the test page, it’s saying it has failed with a timeout after 3.00 seconds. That’s unexpected. Admittedly 3 wall-clock seconds may be too short since this spawns an async to update the local data from another webpage… but it seems to be timing out before that check actually starts, so I’m confused. The log doesn’t seem to provide much information about what timed out.

#2

Is it maybe possible that you have another Skill project that uses the same invocation name and still has your Jovo Webhook URL as endpoint? That happened to me in the past. You could check for this by taking a look at the applicationId field in the request and see if it is the right Skill ID.

You can configure the timeout in the Lambda console https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html#configuration-common-summary

#3

Let’s see. The lambda log says applicationId is “amzn1.ask.skill.987654321”. My local log says it’s “amzn1.ask.skill.c952aa96-df06-4cbf-a9a9-46b7b4113871”. The latter is correct.

However, I only have one Skill project at this time, according to the developer console. Others have existed in the past, but were deleted, so there shouldn’t be crosstalk, right?

I’ve confirmed that I copied the right ARN from the lambda into the skill’s Default and North America regional fields, and that the skill is set to use AWS Lambda.

On a hunch, tried removing the test version of the app from my Alexa account, then re-enabling it for development. Then when I invoked from the device I was getting “There was a problem with the requested skill’s response”. If I invoked from the lambda’s test, I saw the timeout. I guess that’s more consistent.

So I raised the timeout to 3 minutes… and the lambda test succeeded, and didn’t seem to take 20 seconds to do so. And it’s now responding promptly from both the lambda test and the device.

So some combination of those two did the job. Thanks for helping get me refocus…

As far as shortening that timeout goes: I grant that there’s some initialization cost in launching my service, since I’m currently using an in-memory data structure loaded from/saved to a local file. Changing that to a database might help shorten the startup time, and help me lower the worst-case timeout setting. (And/or I need to figure out how to help lambda understand that initial startup takes longer than running once loaded…)

Now I need to figure out how to make jovo deploy automatically update the lambda… The docs suggest that’s possible but it wasn’t immediately obvious to this reader how to configure the Jovo project so it knows which lambda to update. Re-reading…

Having it running as a lambda rather than on my desktop means I can actually consider asking Amazon to make it officially available as a skill. Rah. Still lots to do to polish it, but in theory that can be done incrementally. “Make it work, make it good, make it great… in that order.”

#4

(And I really need to go through and try re-instantiating it from scratch, taking notes and updating my github readme with specific instructions so others can try playing with it.)

#5

Odd behavior: The lambda locked up for a while, reporting an exception and ending silently. (Device’s lights blink the “busy” pattern for a while, then go out, no spoken response.) Just tried the skill again and this time it responded.

  1. Could my having looked at the log been the difference between not responding and responding? I don’t know what else could have caused it to suddenly start working again… unless the other server mine front-ends was offline and has now been restored.

  2. Could Jovo do something more polite to the user when failures occur, eg catching the exception and issuing a standard “try again later” response as other skills do? Or does Jovo not see this happen? (I don’t see a way for me to wrap a try/catch around my whole handler; it appears I’d need to do that explicitly for every intent/event routine. Unless there’s a Javascript trick I haven’t yet learned.)

And a comment: As I suspected, the Lambda runs in a read-only filesystem, so my initial technique of caching the episode table in a local file doesn’t work in that environment. So I’ll have to rework this to use a writable resource, eg a real database. That wants to be a shared resource rather than user specific, so I presume there’s no way to leverage the this.$user.$data db access; I’ll need to open the additional tables directly. I think my incremental load loop suffices to populate a table that starts out empty, and I only need to force a full load if there’s an error during incremental load or I suspect that new data has been added to the master earlier than the incremental scan would find. (Yes, I’m rubber-ducking again, using the process of writing this down to help drive my design. Apologies for the verbosity.)

#6

ISSUE: I’m trying to follow the project.js docs to configure “jovo deploy --target lambda”, but I’m getting a complaint that the user is not authorized to perform lambda:UpdateFunctionCode. It’s showing the username that the lambda should normally run as, which indeed is not authorized to do this.

Theoretically I have an “admin” user which has the needed permission, but it isn’t clear how I tell jovo to deploy using that rather than the execution user. This may be the askProfile: field in project.js, but entering “admin” there doesn’t have the desired effect and I’m not sure what I should provide instead.

Any clarification would be greatly appreciated.

#7

When running ask configure, you should be able to create an AWS profile that is connected to your ASK profile: https://developer.amazon.com/en-US/docs/alexa/smapi/manage-credentials-with-ask-cli.html#ask-configure

#8

Ah. Still figuring out all the different IAM identities, how to create them, what they need, where they’re used… but I seem to have that admin user correctly configured into aws.

PROBLEM:

C:\Users\keshlam\jovo\new-sounds-on-demand>dir bundle.zip
 Volume in drive C is Windows8_OS
 Volume Serial Number is 82F1-EDEF

 Directory of C:\Users\keshlam\jovo\new-sounds-on-demand

11/12/2021  04:00 PM        15,125,124 bundle.zip
               1 File(s)     15,125,124 bytes
               0 Dir(s)  89,747,623,936 bytes free

C:\Users\keshlam\jovo\new-sounds-on-demand>jovo deploy --target lambda

 jovo deploy: Deploys the project to the voice platform.
   >> Learn more: https://jovo.tech/docs/cli/deploy

  √ Bundle Project
    √ Copy source code to "./bundle"
    √ Run "npm install --production"
    ↓ Zip "./bundle" folder [skipped]
      → Zip path: C:\Users\keshlam\jovo\new-sounds-on-demand\bundle.zip
  ↓ Deploying Alexa Skill  [skipped]
    ↓ Uploading to AWS Lambda [skipped]
      → Info: Deployed to lambda function: arn:aws:lambda:us-east-1:04693528706
3…
    √ Enabling skill for testing
  > Deploying Google Action
    ↓ Creating file /googleAction/dialogflow_agent.zip [skipped]
      → Fulfillment Endpoint: undefined
    × Uploading to AWS Lambda
 »   Error: There was a problem:
 »
 »   [ERR] ENOENT: no such file or directory, open
 »   'C:\Users\keshlam\jovo\new-sounds-on-demand\bundle.zip'
 »
 »   Module:   jovo-cli-deploy-lambda
 »

C:\Users\keshlam\jovo\new-sounds-on-demand>

… and sure enough, the zipfile is now missing.

I don’t grok why the bundle zip was skipped (because a file existed?), why upload was skipped during Alexa deploy, why the .zip file is vanishing, or why it’s being tried again during Google deploy (and not skipped). I presume this is not intended behavior, @jan.

Running:
jovo-db-dynamodb : 3.5.3
jovo-db-filedb : 3.5.1
jovo-framework : 3.5.2
jovo-platform-alexa : 3.5.5
jovo-platform-googleassistant : 3.5.4
jovo-plugin-debugger : 3.5.1
aws-cli/1.18.203 Python/3.9.2 Windows/10 botocore/1.19.43
and NPM says everything else is up to date.

#9

Also… When the Lambda service is being spun up, that seems to take long enough that my first request fails with Alexa’s “Sorry, something went wrong” timeout. If I retry several times (or wait a while?) it finally starts responding.

The seems to happen again if the lambda’s been idle for a while, which tends to confirm that the delay is the initialization. As an always-running service that didn’t matter; for lambas which get shut down when idle it impacts response time.

Again, I presume this is not behavior you expect; it would be very confusing/frustrating for end users. Any tips on how to avoid it?

(Amazon’s advice seems to be to use Provisioned Concurrency. But that has costs attached to it. Short of that… adding the environment value AWS_NODEJS_CONNECTION_REUSE_ENABLED=1 may help speed up the DynamoDB and other HTTP(S) requests the lambda issues. See https://pages.awscloud.com/rs/112-TZM-766/images/2020_0316-SRV_Slide-Deck.pdf among others. )

#10

The underlying library that we’re using for the v3 CLI changed a bit, this is why it’s now displaying skipped, although the steps aren’t skipping. This is one of the reasons we fully refactored the CLI (and used a different CLI framework to build it) for v4

#11

The fact that the zip file isn’t there after the run, even if it was before, makes that a bit hard to trust… I can make a change and test that it goes through, I guess, but this really looks like more than a misleading message.

#12

Does it work if you do npm run bundle?

#13

That does create a bundle.zip file:

C:\Users\keshlam\jovo\new-sounds-on-demand>npm run bundle

> [email protected] bundle
> gulp --gulpfile node_modules/jovo-framework/gulpfile.js --cwd ./

[21:46:24] Using gulpfile ~\jovo\new-sounds-on-demand\node_modules\jovo-framework\gulpfile.js
[21:46:24] Starting 'default'...
[21:46:24] Starting 'prepareProject'...

up to date, audited 138 packages in 3s

5 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
[21:46:28] Finished 'prepareProject' after 3.94 s
[21:46:28] Starting 'changePackageFile'...
[21:46:28] Finished 'changePackageFile' after 2.39 ms
[21:46:28] Starting 'build'...
[21:46:41] Finished 'build' after 13 s
[21:46:41] Finished 'default' after 17 s

C:\Users\keshlam\jovo\new-sounds-on-demand>dir *.zip
 Volume in drive C is Windows8_OS
 Volume Serial Number is 82F1-EDEF

 Directory of C:\Users\keshlam\jovo\new-sounds-on-demand

11/17/2021  09:46 PM        15,209,666 bundle.zip
               1 File(s)     15,209,666 bytes
               0 Dir(s)  89,834,909,696 bytes free

Deleting the zipfile and running jovo build --target zip does not seem to be recreating it.

C:\Users\keshlam\jovo\new-sounds-on-demand>dir *.zip
 Volume in drive C is Windows8_OS
 Volume Serial Number is 82F1-EDEF

 Directory of C:\Users\keshlam\jovo\new-sounds-on-demand

File Not Found

C:\Users\keshlam\jovo\new-sounds-on-demand>jovo build --target zip

 jovo build: Build platform-specific language models based on jovo models folder.
   >> Learn more: https://jovo.tech/docs/cli/build

  √ Initializing build process...
    √ Collecting platform configuration from project.js.
      Platforms: alexaSkill, googleAction
    √ Collecting Jovo language model files from ./models folder.
      Locales: en-US
    √ Validating model files.
      √ en-US
  √ Updating Alexa Skill project files
   Path: ./platforms/alexaSkill
    √ Updating Skill Manifest
      Path: ./platforms/alexaSkill/skill-package/skill.json
    √ Updating Alexa Interaction Model
      Path: ./platforms/alexaSkill/interactionModels/custom
      √ en-US
  √ Updating Google Action project files
   Path: ./platforms/googleAction
    √ Updating Dialogflow Agent
      Path: ./platforms/googleAction/dialogflow
      √ agent.json
      √ package.json
    √ Updating Language Model
      Path: ./platforms/googleAction/dialogflow/intents, ./platforms/googleActi
on/dialogflow/entities
      √ en-US

  Build completed.


C:\Users\keshlam\jovo\new-sounds-on-demand>dir *.zip
 Volume in drive C is Windows8_OS
 Volume Serial Number is 82F1-EDEF

 Directory of C:\Users\keshlam\jovo\new-sounds-on-demand

File Not Found

Also repeated:

C:\Users\keshlam\jovo\new-sounds-on-demand>jovo deploy --target=lambda

 jovo deploy: Deploys the project to the voice platform.
   >> Learn more: https://jovo.tech/docs/cli/deploy

  √ Bundle Project
    √ Copy source code to "./bundle"
    √ Run "npm install --production"
    ↓ Zip "./bundle" folder [skipped]
      → Zip path: C:\Users\keshlam\jovo\new-sounds-on-demand\bundle.zip
  ↓ Deploying Alexa Skill  [skipped]
    ↓ Uploading to AWS Lambda [skipped]
      → Info: Deployed to lambda function: arn:aws:lambda:us-east-1:04693528706
3…
    √ Enabling skill for testing
  > Deploying Google Action
    ↓ Creating file /googleAction/dialogflow_agent.zip [skipped]
      → Fulfillment Endpoint: undefined
    × Uploading to AWS Lambda
 »   Error: There was a problem:
 »
 »   [ERR] ENOENT: no such file or directory, open
 »   'C:\Users\keshlam\jovo\new-sounds-on-demand\bundle.zip'
 »
 »   Module:   jovo-cli-deploy-lambda
 »

C:\Users\keshlam\jovo\new-sounds-on-demand>dir *.zip
 Volume in drive C is Windows8_OS
 Volume Serial Number is 82F1-EDEF

 Directory of C:\Users\keshlam\jovo\new-sounds-on-demand

File Not Found

C:\Users\keshlam\jovo\new-sounds-on-demand>

Let me know if there’s any other info I can gather for you.

BTW, one of the suggestions to improve lambda initialization speed is to try compressing/obfuscating the Javascript. Haven’t tried it yet, but it might be worth having --target lambda optionally run something of that sort.

#14

The correct way to create the .zip would be jovo deploy --target zip, iirc.

#15

@rubenaeg: If you only want to create the .zip file, yes.

If you want to automatically upload the zip file, --target lambda (with appropriate configuration in project.js) is supposed to both create it (if necessary, like makefile rules) and uploadl it – or at the very least install an existing zipfile. So the fact that it’s zipping the zip stage isn’t unreasonable if I’ve built the zip file recently.

The fact that it says it’s skipping Alexa and Lambda uploads is an error, though the developers have claimed that’s an error only in the messages.

The error messages during Google deploy, and the fact that the bundle.zip no longer exists at that point, appear to be a functional bug.

C:\Users\keshlam\jovo\new-sounds-on-demand>jovo deploy --target lambda

 jovo deploy: Deploys the project to the voice platform.
   >> Learn more: https://jovo.tech/docs/cli/deploy

  √ Bundle Project
    √ Copy source code to "./bundle"
    √ Run "npm install --production"
    ↓ Zip "./bundle" folder [skipped]
      → Zip path: C:\Users\keshlam\jovo\new-sounds-on-demand\bundle.zip
  ↓ Deploying Alexa Skill  [skipped]
    ↓ Uploading to AWS Lambda [skipped]
      → Info: Deployed to lambda function: arn:aws:lambda:us-east-1:04693528706
3…
    √ Enabling skill for testing
  > Deploying Google Action
    ↓ Creating file /googleAction/dialogflow_agent.zip [skipped]
      → Fulfillment Endpoint: undefined
    × Uploading to AWS Lambda
 »   Error: There was a problem:
 »
 »   [ERR] ENOENT: no such file or directory, open
 »   'C:\Users\keshlam\jovo\new-sounds-on-demand\bundle.zip'
 »
 »   Module:   jovo-cli-deploy-lambda
 »

C:\Users\keshlam\jovo\new-sounds-on-demand>

Hopefully this have been fixed in jovo v4. But while v3 persists, it’d be nice to understand what’s going on here and get it repaired. This worked initially; it’s a bit frustrating to have it break after I’ve seen it succeed.

(I see a new version of v3 is out. I’ll give that a spin.)

#16

Looks like most of the performance problem I was having with lambdas may have been self-inflicted – indexing error between local and UTC timestamps caused my incremental update to reread everything from the back end. Blush.

Not all, alas. My bundle.zip is about 15MB. Spinning up a lambda with this code adds enough delay that Amazon reports my overall p.90 response time as 4.9 seconds and p.50 as 3.7 – really not fast enough for a good user experience.

@jan, is there a best practice for including minify, de-duplication, or other node.js compression/optimization/pruning stages in the development and deployment process? I’m still on Jovo v3, and would rather resolve this before cutting over.

#17

In v4, we drastically improved the build process and provide minification using esbuild. I don’t think this is possible for v3 without architectural changes. Maybe it’s interesting to take a look at the current build scripts in the v4 template package.json

Have you tried increasing the Lambda memory? I heard it has helped with problems like this

#18

Partial solution: I removed a database refresh from startup. When running Jovo continuously, it made sense to do an initial refresh there, to make sure that incremental updates during execution were as short as possible. But in Lambda that just means we’re doing update twice on the first request… and realistically the incremental will generally be fast even without this pre-scan, so the startup synch isn’t especially useful. Making it async doesn’t help, apparently because Lambda waits for startup to complete all threads before it starts responding to the request.

Dropping the pre-synch shaved a few seconds off the first response, bringing the Lambda’s first response down to about a second and Jovo’s response down to about 3. Major improvement.

Lesson: Don’t do any work in lambda startup that you’re going to redo in lambda response, and consider that lambda setup may be effectively synchronous as seen by lambda even if the javascript inside it is running async.

(Longer term, I’m looking at moving from my in-memory database cache to DominoDB, so it can be left set up and updated on a scheduled basis completely separately from the skill’s lambda.)

#19

In production, the changes have been:

  1. Yes, use DynamoDB as my data store.

  2. Move the synchronization process out of the Jovo skill and into a second lambda, triggered by a cron scheduled event at specific times of day. That takes the update time completely out of the skill’s response time.

The user may still see a delay as the lambda spins up, if it hasn’t been used for a while – but response is now reliably under a second even with that overhead; MUCH more acceptable.