We are nearing completion for “alpha” for the project I discussed at serverlessconf.io. I have started doing stress and loading testing for some scenarios. Some observations:
- Load Impact is an excellent tool. The product is extremely easy to use. Defining and configuring tests is quick, intuitive and easy. I can build complex test scenarios in Lua with a small subset of the language. The Load Impact library is complete and simple to use. I was initially not thrilled to learn a new language, but Lua is extremely interesting and is also the language for extending Redis. Redis, especially in AWS, is core technology for scaleable apps.
- There was a startup learning curve for AWS Lambda, which our use of Java compounded. There was an initial, one time productivity hit relative to using technology compared to alternatives with which we have experience (e.g. Tomcat). This has been more than offset by the total absence of work necessary to define, configure, operate and monitor server technology for unit test, integration test and load/stress testing. We have done no configuration or operation, and use the same logic environment for unit test, integration test and stress testing. We just change some parameters in Load Impact, and the same client config drives the same “server” config. My guess is that these capabilities are increasing our development efficiency by at least 20%. Our IT dev, test and delivery “hardware” is basically our laptops. Everything else is serverless.
- We have amused many people with our use of “archaic” technology, specifically Java and relational databases (MySql). Mark Boyd’s article on the serverless conference correctly observed that the majority of serverless projects are startups or leading edge, or in areas adjacent to core web applications/business logic (e.g. image, cognition). Java and RDB support provide an onramp for more traditional enterprise app developers, and the development efficiency we observed should accelerate enterprise adoption.
The “self-tuning” nature of AWS Lambda is impressive (see figure). The scenario ramped from 0 to 500 users in 5 minutes. We got some 504 errors (gateway timeout) during the ramp up. This is the first time we saw these errors, because previous tests ramped up over 10 minutes. The yellow line is failure rate, which quickly goes to zero. Req/s tails off at the end because the test is shutting down. RT and total virtual user scenario time relatively quickly stabilizing after some bumpiness.
All of this indicates significant productivity and optimization as we start going into application delivery.