0

Integrate with Ease – AWS + Twilio, Slack and IoT

Amazon Web Services (AWS) is changing the way engineers develop solutions.  It is so easy to prototype and have a scalable architecture with very little hand holding from the Office of the CTO or Systems Engineers.  This, in turn, fosters the DevOps culture within the organization.

One of the prototypes we’ve tried is to integrate AWS with Twilio, Slack and Intel Edison + Grove IoT device.  We cannot take all the credit here because this was inspired by a recent trip to AWS’s San Francisco pop-up loft.  They had a zombie apocalypse themed workshop but we took it a step further and used our company use cases and dissected each step since when we were there, we were just going through the motions.  We also thought that learning about how some of these technology companies work will provide us a fresh new perspective on what is moving and shaking the software industry.

The discussion below assumes some AWS knowledge of some of the services used.  It does not elaborate on each service and explain it in detail but here’s a quick refresher taken from AWS themselves:

  1. Lambda – “Lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running”
  2. API Gateway – “Makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale”
  3. SNS – “Pub-sub Service for Mobile and Enterprise Messaging”
  4. Dynamo DB – “Fast and flexible NoSQL database service”

Let’s get right to it, shall we?  First up…

Twilio

Twilio is a technology company that allows programmable communications.  They are the interface for sending and receiving global SMS and MMS messaging from any app.

Who uses Twilio?

  1. Box
  2. Nordstrom
  3. OpenTable
  4. Intuit
  5. Uber
  6. EMC2
  7. Zendesk
  8. Cocacola

Main competitors:

  1. Nexmo
  2. Plivo

Use Case:

Our use case is to send a client their next scheduled appointment to speak with one of our investment advisors if they send a text message of “schedule”.  For this prototype, the date would have been previously set.

Solution Architecture

twilio

Components:

User-appt-schedule is the API gateway that fronts the lambda function named user-schedule-get.  The lambda function then fetches the data from the dynamo DB for the schedule date.

AWS:

  1. Create your Dynamo DB with the primary key of phone number and column for appointment date.pic2
  2. Create your node JS lambda function with that gets the date from the DB given the phone number.

console.log('Loading function');
var aws = require('aws-sdk');
var ddb = new aws.DynamoDB(
{region: "us-west-2",
params: {TableName: "participant-schedule-reminder"}});

var theContext;

function dynamoCallback(err, response) {
if (err) {
console.log('error' + err, err.stack); // an error occurred
theContext.fail(err);
}

else {
console.log('result: ' + JSON.stringify(response)) // successful response
console.log('parsed response ' +JSON.stringify(response.Item.scheduledDate.S).replace("\"", ""));
theContext.succeed("Your next retirement checkup is scheduled on " +
JSON.stringify(response.Item.scheduledDate.S).replace("\"", ""));
}
}

exports.handler = function(event, context, callback) {
theContext = context;
console.log("Request received:\n", JSON.stringify(event));
console.log("Context received:\n", JSON.stringify(context));

//Determine text
var textBody = event.body;

//Get the phone number, only the last 10 characters
var phoneNumber = event.fromNumber.substring(2, 12);
console.log('phone ' + phoneNumber);

if (event.body.trim() == "schedule"){

var params = {
Key:{
"phone-number": { N: phoneNumber }
},
AttributesToGet: [
"scheduledDate"
],
}
//var response = ddb.scan(params,dynamoCallback);
var response = ddb.getItem(params,dynamoCallback);
} else {
theContext.succeed("Text 'schedule' to get your retirement check up date");
}
};

3.  Create your API gateway that invokes the lambda function for each get request.  The nuance with this is that every request that comes from Twilio is in TwiML format (XML format) but our lambda function requires a JSON object so a conversion has to be done in the integration piece of the API gateway and every JSON response from the lambda function needs to be converted to TwiML format.

3

4

Twilio:

  1. You can sign up for a trial account with Twilio and they you will get a phone number from them.
  2. You can then program this phone number with web hooks.  The URL below is the ARN URL from AWS for the API gateway.

6

Slack

Slack is an instant messaging and collaboration system.  They have teams and channels and have robust features for search.  There are commands that can be programmed called “slash commands” which allows for forwarding the message typed in the chat box to an external source with the use of a web hook.

8

Who uses Slack?

  1. Airbnb
  2. CNN
  3. Buzzfeed
  4. EA Sports
  5. Ebay
  6. Harvard University
  7. Samsung
  8. Expedia
  9. Intuit

Main Competitors:

  1. HipChat
  2. Yammer
  3. Google Hangouts
  4. Facebook at work

Use Case:

Our use case is to send a client their next scheduled appointment to speak with one of our investment advisors if they send a slash command in the appropriate FE channel.  The date would have been previously set.

Solution Architecture:

9

Slack:

  1. Create a slash command (“fngn” in this example).  You still wouldn’t have the URL at this point since that will come from AWS.  The token below will be copied over to AWS’ lambda function to verify that the request came from the right channel.

10

AWS:

  1. Create a Dynamo DB with the slack handle as the primary key:
    11
  2. Create a lambda function that takes the handle and looks it up in the DB.
  3. Create an API gateway to convert the XML format from slack to JSON and vice versa.

12

Intel Edison + Grove IoT device

Intel Edison is a chip where code can be pushed built for wearables and Internet of Things (IoT) devices.  The grove toolkit contains a variety of widgets that can be attached to the Edison board.

13

Use case:

When motion is detected from the IoT device, send me an email notification.

Solution Architecture:

14
Grove has a motion detector widget so that was attached to the Edison board.

  1. Intel XDK IoT Edition is the IDE that provides templates to create Nodejs projects and deploy the code for sensors to Edison.
    16
  2. From there, it’s just a matter of setting up the SNS service so that every motion detected triggers a publish to the email.
    17

Conclusion

This was a very fun hack.  Not only did I find out more information about Twilio, Slack and IoT but it made me realize how easy it is to prototype and possibly productize solutions using the power of AWS.  We are now only limited by the power of our imagination.

4

How we improved our EV charging station sharing with HipChat, AWS and ChargePoint API

Have you ever noticed that there are never enough EV charging stations at work?

You probably have if you own an electric car or a hybrid. Electric vehicles and plug-in hybrids have come a long way and are steadily growing in popularity. A combination of government incentives and exemptions from carpool lane rules make them a great choice for commuters. As a result, it is a common employee perk in Silicon Valley, for a company to offer free or subsidized charging at work. However, it seems the number of available charging stations always seems to be dwarfed by the growing number of electric car drivers. Even if there are several chargers, the process around sharing the chargers between employees is never perfect and someone inevitably gets stuck without enough juice to drive home.

At Financial Engines Sunnyvale, CA headquarters, we have four charging stations managed by ChargePoint Network. At the time of this writing, there are roughly 35 EV drivers trying to get a charge on a daily basis. At first, our “sharing” process didn’t quite work. First, there was no visibility into when people plugged in and out. Our offices in Sunnyvale are far enough from the charging stations that you cannot see if there is one available at any given point. Second, not everyone bought into the whole “sharing” idea, and sometimes cars would continue to occupy the precious charging spots long after their car was fully  charged.

EV Concierge to the rescue

By experimenting with various options, and with a little bit of coding and iterating, we developed a nifty system that is fun to use and works much better – so much so that on most days, by late afternoon we have chargers available and nobody is questioning that “sharing” concept anymore. We call it – EV Concierge.

EV Concierge is an automated assistant that monitors our charging session availability, manages orderly queue of drivers who need a charge and occasionally shames you into moving your car in time for others to get a charge.

How it works

EV Concierge2

At the center of the system, we have HipChat, which is already used by nearly everyone in the company for daily collaboration. We set up a dedicated room, called “EV Charging” and that is where all the magic takes place.

It all starts with an employee typing an “ev add” command to put themselves in the queue. He (let’s call him Dave) does it right there in the HipChat window, the same way he would post a short message. In response, the system will put Dave at the bottom of the queue and display a complete listing of the queue in the same HipChat window. Now Dave knows where he is on the list.

In the background, we have a monitoring process running (EV Concierge), which is communicating with ChargePoint network via XMPP protocol and listens as people plug-in and plug-out. In response to these events, EV Concierge notifies everyone in the room about a change in charger availability and also informs the next person in the queue that it is his or her turn. That is how Dave knows that it is his turn. EV Concierge will automatically remove Dave from the queue as soon as it detects that Dave has actually plugged in.

Growing feature list

Adding new features to EV Concierge and using it daily has been equally  fun. There is something to be said about being able to interact with your customers and iterate on the solution on a daily basis. Iterative development does not get better than that! Since basic functionality was put in place, we enhanced EV Concierge with a number of useful features. Here is a complete list:

  1. ev add/remove/list – driver queue management
  2. ev suspend/resume – keeps your place in the queue but lets person behind you go ahead of you. This is useful when you are stuck in a meeting.
  3. ev next – calls out next person in the queue. This happens automatically when charger becomes available.
  4. Reminders:
    • When a charger is available for more than 10 minutes, EV Concierge will notify the room about precious time being wasted.
    • When a user has been charging for more than 2.5 hours, EV Concierge will (politely) ask him or her to move their car to let others partake in the experience.

In fact, EV Concierge has been getting so smart, people sometimes talk back to it, not realizing it is a robot. 🙂

How we built it

If you are intrigued by this creative solution to a common workplace problem, you can build your own EV Concierge relatively easily. We are working to open source our implementation, but in the meantime, here is how to do it.

ChargePoint API

Charge Point publishes their Web Services API and it can be found here https://na.chargepoint.com/UI/downloads/en/ChargePoint_Web_Services_API_Guide_Ver4.1_Rev4.pdf

If your company owns the chargers, most likely you have a support agreement with ChargePoint and your facilities manager has the API key that you will need to integrate with the API. There are two ways to talk to ChargePoint: SOAP API and XMPP protocol. We use a combination of both because they expose different levels of information about drivers and their sessions. In fact, the level of detail you get in the API is highly dependent on your corporate support agreement with ChargePoint and will control how sophisticated of a system you can build. If you are a Java developer, Smack API is an easy to use library for XMPP listener.

HipChat Integration

There are many ways to integrate with HipChat. For on-premise installations, you can extend HipChat with HUBOT. It is a tiny process that can listen for special words and execute commands. It uses CoffeeScript and is relatively easy to use. For cloud versions, you can use a feature called “slash” which allows you to map anything that starts with a “slash” (duh!) to a REST API call. Finally, to programmatically post messages to HipChat from EV Concierge, you can use the simple REST interface that HipChat supports out of the box. For extra fun, you can extend HipChat with custom icons and make your EV Concierge messages include some branding (or character).

If your company uses Slack, Yammer, or Google Hangouts instead of HipChat – no worries. The same type of integration can be achieved with any of these systems. The key is to try not to introduce yet another communications interface in your workplace. If you want high adoption, stick with existing tools.

Queue Persistence

This can be done in a thousand different ways, but if you have access to the AWS ecosystem, it is an obvious choice. You can use a document storage DB like AWS Dynamo, put the persistence logic in an AWS Lambda function, slap a REST interface on top of it through AWS API Gateway and be done in a day or less. This was a hackathon project, so I know for a fact that it can be done in less than a day, even if you have never heard of Lambda or Dynamo before.

EV Concierge

Finally, the heart of the logic is implemented as a standalone Java program. It loads the initial charger status via ChargePoint SOAP API on start up and then goes into listening mode for events. With every event, it keeps track of who is plugged in, who is finished and who should be notified. To make HipChat messaging personal, you will need to map ChargePoint user ids that you get through the API to your internal HipChat user ids. This will allow you to automatically manage the queue when a driver is recognized and also send direct messages using HipChat “@” feature.

EV Concierge is deployed on Amazon cloud via Elastic Beanstalk, which simplifies deployment and management of the process. One caveat to note is that we noticed that XMPP connection gets stale after prolonged use (12+ hours), so we had to build an automated restart at 6AM every morning to renew it. It works fine for us because EV Concierge can sleep at night ahead of a busy working day.

Conclusion

EV Concierge has been a fun little project to work on. It’s great to have a problem that is so well defined and actionable that it begs to be solved. It’s also a great feeling to build something that your fellow colleagues can use and enjoy on a daily basis. As I mentioned, since we adopted EV Concierge, many more drivers get a chance to charge and the process is much smoother and more fun for everyone. The list of proposed features is also steadily growing. Oh, if one can only find enough time…

As part of the hackathon, we put together a little presentation that describes EV Concierge in 3 minutes and shows it in action. Here is the video for your to enjoy.