Project 7: Story Time with AT&T Enhanced WebRTC

June 25, 2015 |

The other day I went to AT&T’s Devlab at Plug and Play in Sunnyvale. It was almost by chance, having seen the event page the day before. And who would turn down an offer for a free lunch and free premium access to their Enhanced WebRTC API. I’m so gullible to free stuff!

The three hour event consisted of about seven labs, each progressively adding additional buttons and functionality to a phone app demo. The labs covered setting up the Node.js backend, logging in as a user, making a call to another user, receiving a call, holding and resuming a call, and muting and unmuting a call.

The labs were perfectly paced for a beginner and were very straightforward. I ended up blasting through the labs really quickly, and finished early. Still thinking up an idea for Project 7, I was inspired by the video part of WebRTC.

For this seventh project of my 15 project in 30 days challenge, I’m going to convert some of the sample code from the labs into AngularJS, and then use WebSockets to add a story book where the pages can be flipped through, and with the other user’s story book changing as well.

This project is inspired by a story I heard from a friend. As a parent who travels for work, he misses bedtime stories with his kids. When possible, he uses Apple’s FaceTime to connect with his kids to read them a story. It works with some compromises.

AT&T Enhanced WebRTC

It is probably easier to get the AT&T Enhanced WebRTC Node.js example and start from there. Follow the Getting Started, Create an App, Using the SDK, and Sample App Deployment instructions on AT&T’s SDK Quick Start guide. If you want to play around a little bit, this set of examples looks pretty close to what the Devlab covered.

By the way, I just noticed this additional resource that gives a better understanding of the SDK. It could have helped me.

Setup

I used the Node.js example and the att7.html file as a base for this project. I renamed att7.html to storytime.html, added storytime.js (AngularJS controller), and modified the app.js to handle WebSockets. Here are the specific changes:

<!-- Filename: public/storytime.html -->
<!DOCTYPE html>
<html ng-app="storyTimeApp">
<head lang="en">
  <meta charset="UTF-8">
  <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.3.16/angular.min.js"></script>
  <script src="storytime.js"></script>
  <script src="/socket.io/socket.io.js"></script>
</head>
<body ng-controller="storyTimeCtrl" style="font-family:Arial">
  <h2>Story Time</h2>
  <div ng-show="status.isConnected" style="float:right; width:300px; border:1px solid black; padding:20px; text-align:justify">
    <button ng-click="nextPage()" ng-disabled="page == 3" style="float:right">Next Page</button>
    <button ng-click="previousPage()" ng-disabled="page == 0">Previous Page</button>
        
    <hr />
    <div ng-show="page == 0">
      <h2>CHAPTER I. Down the Rabbit-Hole</h2>
      <p>Alice was beginning to get very tired of sitting by her sister on the
        bank, and of having nothing to do: once or twice she had peeped into the
        book her sister was reading, but it had no pictures or conversations in
        it, 'and what is the use of a book,' thought Alice 'without pictures or
        conversations?'</p>
      <p>So she was considering in her own mind (as well as she could, for the
        hot day made her feel very sleepy and stupid), whether the pleasure
        of making a daisy-chain would be worth the trouble of getting up and
        picking the daisies, when suddenly a White Rabbit with pink eyes ran
        close by her.</p>
    </div>
    <div ng-show="page == 1">
      <p>There was nothing so VERY remarkable in that; nor did Alice think it so
        VERY much out of the way to hear the Rabbit say to itself, 'Oh dear!
        Oh dear! I shall be late!' (when she thought it over afterwards, it
        occurred to her that she ought to have wondered at this, but at the time
        it all seemed quite natural); but when the Rabbit actually TOOK A WATCH
        OUT OF ITS WAISTCOAT-POCKET, and looked at it, and then hurried on,
        Alice started to her feet, for it flashed across her mind that she had
        never before seen a rabbit with either a waistcoat-pocket, or a watch
        to take out of it, and burning with curiosity, she ran across the field
        after it , and fortunately was just in time to see it pop down a large
        rabbit-hole under the hedge.</p>
      <p>In another moment down went Alice after it,</p>
    </div>
    <div ng-show="page == 2">
      <p>never once considering how in the world she was to get out again.</p>
      <p>The rabbit-hole went straight on like a tunnel for some way, and then
        dipped suddenly down, so suddenly that Alice had not a moment to think
        about stopping herself before she found herself falling down a very deep
        well.</p>
      <p>Either the well was very deep, or she fell very slowly, for she had
        plenty of time as she went down to look about her and to wonder what was
        going to happen next. First, she tried to look down and make out what
        she was coming to, but it was too dark to see anything; then she looked 
        at the sides of the well, and noticed that they were filled with
        cupboards and book-shelves; here and there she saw maps and</p>
    </div>
    <div ng-show="page == 3">
      <p>pictures hung upon pegs. She took down a jar from one of the shelves as
        she passed; it was labelled 'ORANGE MARMALADE', but to her great
        disappointment it was empty: she did not like to drop the jar for fear
        of killing somebody, so managed to put it into one of the cupboards as
        she fell past it.</p>
    </div>
  </div>

  <p><span>{{statusLabel}}</span></p>

  <p ng-hide="status.isLoggedIn"><input type="text" ng-model="username">@{{att.ewebrtcDomain}}</p>

  <p>
    <button ng-hide="status.isLoggedIn" ng-click="login()">Login</button>
    <button ng-show="status.isLoggedIn" ng-click="logout()">Logout</button>
  </p>

  <p ng-show="status.isLoggedIn">
    <input ng-show="!status.isConnected" type="text" ng-model="callTo" placeholder="Account ID">
    <button ng-show="!status.isConnected" ng-click="makeCall()">Make Call</button>
    <button ng-show="status.isConnected && !status.isOnHold" ng-click="holdCall()">Hold Call</button>
    <button ng-show="status.isConnected && status.isOnHold" ng-click="resumeCall()">Resume Call</button>
    <button ng-hide="!status.isConnected" ng-click="endCall()">End Call</button>
    <button ng-show="status.isIncomingCall" ng-click="answerCall()">Answer Call</button>
    <button ng-show="status.isConnected && !status.isOnHold && !status.isMuted" ng-click="muteCall()">Mute Call</button>
    <button ng-show="status.isConnected && !status.isOnHold && status.isMuted" ng-click="unmuteCall()">Unmute Call</button>
  </p>

  <p ng-show="status.isConnected"> 
    <video id="remoteVideo" width="320" height="240"></video>
    <video id="localVideo" width="64" height="48"></video>
  </p>

  <div style="clear:both; padding-top:10px">
    <hr />
    Adapted from Enhanced WebRTC DevLab by AT&T Developer Program.
  <script src="//code.jquery.com/jquery-2.1.1.min.js"></script>
  <script type="text/javascript" src="/js/ewebrtc-sdk.min.js"></script>   
</body>
</html>

// Filename: public/storytime.js

angular.module('storyTimeApp', [])
.controller('storyTimeCtrl', function($scope) {
  $scope.att = {
    myDHS: 'http://0.0.0.0:9000',
    ewebrtcDomain: '',
    appTokenUrl: '',
    accessToken: {},
    phone: ATT.rtc.Phone.getPhone()
  };
  $scope.status = {
    isLoggedIn: false,
    isIncomingCall: false,
    isConnected: false,
    isOnHold: false,
    isMuted: false
  };
  $scope.socket = io();
  $scope.statusLabel = '';
  $scope.username = '';
  $scope.callTo = '';
  $scope.room = '';
  $scope.page = 0;
  
  $scope.socket.on('message', function(message)
  {
    console.log(message);
  });

  $scope.onConfig = function(data) {
    $scope.$apply(function() {
      $scope.att.ewebrtcDomain = data.ewebrtc_domain;
      $scope.att.appTokenUrl = data.app_token_url;
      console.log($scope.att);
    });
  }

  // Get the domain app token URL
  $.ajax({    
    type: 'GET',
    url: $scope.att.myDHS + '/config',
    success: $scope.onConfig,
    error: $scope.onError   
  });

  $scope.onError = function(data) {
    console.log('onError');
    $scope.$apply(function() {
      $scope.statusLabel = 'Error: ' + data.error.ErrorMessage;
    });
  }

  $scope.onLogin = function(data) {
    $scope.$apply(function() {
      $scope.att.accessToken = data;
      $scope.associateAccessToken();
    });
  }

  $scope.login = function() {
    // Obtain access token

    $.ajax({    
        type: 'POST',
        url: $scope.att.appTokenUrl, 
        data: {app_scope: 'ACCOUNT_ID'},
        success: $scope.onLogin,
        error: $scope.onError
    });   
  }

  $scope.onAssociateAccessToken = function() {
    $scope.att.phone.login({token: $scope.att.accessToken.access_token});
  }

  $scope.onLoginSuccess = function(data) {
    $scope.$apply(function() {
      $scope.statusLabel = 'Login successful';
      $scope.status.isLoggedIn = true;
      console.log(data);
    });
  }

  $scope.associateAccessToken = function() {
    $scope.att.phone.associateAccessToken({
      userId: $scope.username,
      token: $scope.att.accessToken.access_token,
      success: $scope.onAssociateAccessToken,
      error: $scope.onError
    }); 
  }

  $scope.onLogout = function() {
    $scope.$apply(function() {
      $scope.statusLabel = 'Logged out';
      $scope.status.isLoggedIn = false;
    });
  }

  $scope.logout = function() {
    $scope.att.phone.logout();
  }

  $scope.makeCall = function() {
    $scope.statusLabel = 'Calling '+$scope.callTo;

    $scope.att.phone.dial({
      destination: $scope.att.phone.cleanPhoneNumber($scope.callTo),
      mediaType: 'video',
      localMedia: document.getElementById('localVideo'),
      remoteMedia: document.getElementById('remoteVideo')
    });
  }

  $scope.endCall = function() {
    $scope.att.phone.hangup();
  }

  $scope.onCallConnected = function(data) {
    $scope.$apply(function() {
      $scope.status.isIncomingCall = false;
      $scope.room = (data.hasOwnProperty('to')) ? 
        'sip:'+$scope.username+'@'+$scope.att.ewebrtcDomain+':sip:'+data.to :
        data.from+':sip:'+$scope.username+'@'+$scope.att.ewebrtcDomain;   
      
      console.log('room:'+$scope.room);
      $scope.socket.emit('join', $scope.room);


      console.log(data);
      $scope.statusLabel = 'Call connected with '+(data.from || data.to);
      $scope.status.isConnected = true;
    });
  }

  $scope.onCallDisconnected = function(data) {
    $scope.$apply(function() {
      $scope.statusLabel = 'Call disconnected';
      $scope.status.isConnected = false;
      console.log('leaving room:'+$scope.room);
      $scope.socket.emit('leave', $scope.room);
      $scope.room = '';
    });
  }

  $scope.answerCall = function() {
    $scope.att.phone.answer({
      mediaType: 'video',
      localMedia: document.getElementById('localVideo'),
      remoteMedia: document.getElementById('remoteVideo')
    });  
  }

  $scope.onIncomingCall = function(data) {
    $scope.$apply(function() {
      $scope.status.isIncomingCall = true;
      $scope.statusLabel = 'Incoming call from '+data.from;
    });
  }

  $scope.holdCall = function() {
    $scope.att.phone.hold();
  }
      
  $scope.resumeCall = function() {
    $scope.att.phone.resume();
  } 

  $scope.onHeldCall = function() {
    $scope.$apply(function() {
      $scope.statusLabel = 'Call on hold';
      $scope.status.isOnHold = true;
    });
  }        

  $scope.onResumedCall = function() {
    $scope.$apply(function() {
      $scope.statusLabel = 'Call resumed';
      $scope.status.isOnHold = false;
    });
  } 

  $scope.muteCall = function() {
      $scope.att.phone.mute();
  }
      
  $scope.unmuteCall = function() {
      $scope.att.phone.unmute();
  }  

  $scope.onMutedCall = function() {
    $scope.$apply(function() {
      $scope.statusLabel = 'Call muted';
      $scope.status.isMuted = true;
    });
  }        

  $scope.onUnmutedCall = function() {
    $scope.$apply(function() {
      $scope.statusLabel = 'Call unmuted';
      $scope.status.isMuted = false;
    });
  } 

  $scope.nextPage = function() {
    $scope.page++;
    $scope.statusLabel = 'Changing to page '+$scope.page;    
    $scope.socket.emit('update', {room: $scope.room, event: 'page:changed', page: $scope.page});
  }

  $scope.previousPage = function() {
    $scope.page--;
    $scope.statusLabel = 'Changing to page '+$scope.page;    
    $scope.socket.emit('update', {room: $scope.room, event: 'page:changed', page: $scope.page});
  }

  $scope.att.phone.on('error', $scope.onError);  
  $scope.att.phone.on('session:ready', $scope.onLoginSuccess);
  $scope.att.phone.on('session:disconnected', $scope.onLogout);  
  $scope.att.phone.on('call:connected', $scope.onCallConnected);
  $scope.att.phone.on('call:disconnected', $scope.onCallDisconnected);
  $scope.att.phone.on('call:incoming', $scope.onIncomingCall);
  $scope.att.phone.on('call:held', $scope.onHeldCall);
  $scope.att.phone.on('call:resumed', $scope.onResumedCall);
  $scope.att.phone.on('call:muted', $scope.onMutedCall);
  $scope.att.phone.on('call:unmuted', $scope.onUnmutedCall);

  $scope.socket.on('page:changed', function(data) {
    console.log(data);
    $scope.$apply(function() {
      $scope.statusLabel = 'Changing to page '+data.page;
      $scope.page = data.page;
    });
  });
});

In app.js, replace the two lines:

  http.createServer(app).listen(http_port);
  console.log('HTTP web server listening on port ' + http_port);

with

  var server = http.createServer(app);
  var io = require('socket.io').listen(server);
  server.listen(http_port);

  io.on('connection', function(socket) {
    console.log('a user connected');

    socket.on('join', function(room) {
      socket.join(room);
      console.log('joining '+room);
    });

    socket.on('leave', function(room) {
      socket.leave(room);
      console.log('leaving '+room);
    });

    socket.on('update', function(msg) {
      socket.broadcast.to(msg.room).emit(msg.event, msg);

      console.log('sending room "'+msg.room+'" message: '+JSON.stringify(msg));
    });    
  });  

  console.log('HTTP web server listening on port ' + http_port);

In package.json, I had to change the following to host the app on my webserver:

Added "0.0.0.0:9000" and "0.0.0.0:9001" to cors_domains (change 0.0.0.0 to your IP address)
Changed sandbox.app_key and sandbox.app_secret to my App Key and App Secret from the developer console.
Changed oauth_callback, app_token_url, app_e911id_url, to point to my server IP address
Changed ewebrtc_domain to the domain I chose in the developer console
Under dependencies, added "socket.io": "^1.3.5",

To install the dependencies:

npm install

To run the Node.js app, run the following command:

npm start

Story Time

If all goes well (I know there are a lot of pieces to this project), you should be able to access the app at:

http://IPADDRESS:9000/storytime.html

You’ll need two Chrome browser windows open on separate computers (side by side on one computer, like I did, could work, but the videos will all use the same webcam). In one window, log in as dad. In the other, log in as son.

Next, dad should call the son. Enter son@{{domain}} and click on Make Call.

An Answer Call button appears on the son’s screen. Click on Answer Call.

After a second or two, videos should be up on both sides.

You might be wondering why all four videos look the same. I’m using one computer, one webcam, and two browsers. The next two screenshots have been edited so it’s clearer who’s the dad (the mouth is open telling a story) and who’s the son (smiling, excited for a bedtime story).

Back to the demo. Now that we have the video part working, thanks to the AT&T Enhanced WebRTC example, you’ll also notice a book on the right side has appeared.

As the dad reads the book on the right, he can navigate to the next page using the Next Page button. This will send a message through the WebSocket to the son’s browser, where it will be processed by the WebSocket listener.

When it’s time for bed. They can click on End Call to end the call. The End. Don’t let the bedbugs bite.

WebSocket

I want to talk a little more about the WebSocket piece. When a call is connected, the from SIP address value is concatenated with the to SIP address value, and is saved into $scope.room. This value is used to join a room on the socket.io connection. This room is a separate “channel” just for this call, instead of the general room everyone is part of and where everyone receives any messages sent in the generic room. Both parties send a message via the general socket to join this room, send messages to just this room, and then leaves this room on call disconnect.

When someone presses the Next Page or Previous Page, a message is sent to the WebSocket with a page:changed event and the page number. The other listeners in this room receive this message, changes the $scope.page value, which AngularJS automatically shows (via data-binding) the div corresponding to the book page content.

With all that said, it would have been easier if there was a data channel between participants on the WebRTC connection, but I wasn’t able to find anything to make this happen.

Source Code

You can find the repo on GitHub.

Expansion

That’s it for this project. Here are some additional ideas on how this project can be expanded:

I came across the conference examples, but didn't get a chance to see if adding a third party (grandma always likes a good story) would be feasible.
The story content could include images which makes noise when clicked on. Send a message across the WebSocket so the other person can hear it as well.

## Post-mortem

This project was particularly challenging because of all the individual parts. There’s the sample project from the Devlab, partially rewritten into AngularJS, and then extended with WebSockets to add additional functionality. It could definitely be refactored and cleaned up a bit, but I chose to extend the sample project instead of butchering it.

I could have also used PubNub instead of WebSockets to manage the page turning.

15 Projects in 30 Days Challenge

This blog post is part of my 15 projects in 30 days challenge. I’m hacking together 15 projects with different APIs, services, and technologies that I’ve had little to no exposure to. If my code isn’t completely efficient or accurate, please understand it isn’t meant to be complete and bulletproof. When something is left out, I try to mention it. Reach out to me and kindly teach me if I go towards the dark side. ?

This challenge serves a couple of purposes. First, I’ve always enjoyed hacking new things together and using APIs. And I haven’t had the chance (more like a reason) to dive in head first with things like AngularJS, NodeJS, and AT&T Enhanced WebRTC. This project demonstrated AngularJS, Node.js, and AT&T Enhanced WebRTC.