Engineering

Transforming the Web User Interface

The Turn Console, the gateway to the Turn Platform, lets marketers run advertising campaigns and interpret meaningful insights that drive business decisions and strategy. Just one misaligned number or incorrect behavior on a button could prove a costly mistake, meaning the user interface is absolutely critical.

When Turn was founded, web technologies did not revolve around JavaScript. Turn originally used Apache Velocity as the templating engine for the view layer powering the Turn Console. These templating engines served the purpose of creating a UI that actually separated the View (Velocity) and Model (Java objects), but the amount of interactions were limited. User interactions in Velocity templated pages are more of an afterthought than first-class citizen. Scripts were added later to make the page interactive. From the development perspective, Velocity was also difficult to debug, to pinpoint issues, and to rapidly develop on. For a team that was growing quickly, Velocity lacked tools and modularization for us to properly scale. So much has changed over the past 10 years. What was trendy and worked back then may not work by today’s standards. This is especially true for front end (UI) stack, the code that runs inside browsers, or the “view” from the standard MVC (Model-View-Controller) paradigm.

After JavaScript started to dominate the web as we know today, the velocity of this evolution has been epic. With better JavaScript engines in browsers, many of the traditional logic of processing and rendering of the page elements can be shifted to the client side. Single-Page Application came into the picture. Single-Page Application relies heavily on JavaScript to render and make changes to the DOM elements in the browser. The advantages of using the Single-Page Application includes but are not limited to the following:

  • No full-full page refreshes so it feels more responsive and interactive

  • Only the data portion of the refreshed page needs to be re-sent reducing bandwidth

  • Easier to pinpoint problems and debug

  • Easier to test with mocked endpoints

  • Lets the user feel it is a web application rather than a web page

  • Development tools to make it easier to build, distribute and scale

  • API’s come as the byproduct of endpoints we build for the application

  • Better modularization of UI components

Several points highlighted above caught our attention when we started to explore options to revamp the Turn Console. We wanted to enhance the user experience, performance, maintainability as well as scalability of the Turn Console. We have a few options:

  1. Make a homegrown Single-Page Application solution that is rendered on the client side

  2. Pick a front-end framework that works well with Single-Page Application

Moving this large web application is already a big task and we did not want to add the overhead of designing, implementing and maintaining a homegrown framework on top of that. A homegrown solution would not allow a newcomer to ramp up quickly due to lack of previous experience and lack of available tutorials. We decided an existing framework would better suit our needs.

We wanted an UI framework will offer modularization of UI components. We also needed tools to support a modern web application and a growing team. We evaluated a few options including Ember.js, Backbone, and AngularJS. We wanted a framework that can offer project structure, flexibility, a templating engine, a test framework, and good community support. Backbone is the lightest of all the frameworks we evaluated. We felt that Backbone was just a thin wrapper on top of jQuery and it did not offer too much of a framework per se. Ember.js favored convention over configuration. Given that we are not a Ruby shop, we liked to have explicit configuration. Furthermore, Ember is more of a collage of frameworks than a single framework that is built from the ground up. We ended up picking Angular for the job. When we started using Angular, it was still early days of Angular and we had doubts initially especially since Angular tampered with basic html to add its own tags and attributes which seemed a bit odd at the time. We tested out Angular on a few pages we wanted to migrate. Our doubts quickly went away after we witnessed the ease of development.

Angular’s dependency injection, directives, services, modularization made our UI code more structured and cleaner. Directives and services eliminated much of duplicated code that existed in our legacy code. Turn directives included an interactive and infinite scroll list table as well as well as a timezone supported datepicker. After these components are created, it is just plug and play for the rest of the pages. Dependency injection also made unit testing and mocking easier. The modularization assured us that namespace and variable collision will not be an issue anymore. Controllers in angular wrapped the most of the complex logics we have in our console; however, each of these controller were self contained and the complexity were not spilled over into another page. Being able to use any language to generate the JSON endpoints made development extremely easy. We were able to use NodeJS servers to mock the endpoints we wanted as opposed to create the data we needed in the database. Proxy servers were also created to feed data from staging or production server to your local development set quickly.

In terms of user interaction, Angular also made it possible to limit the amount of full page refreshes that is done. This makes the pages more responsive and seemingly quicker to load. With updating our CSS library to Bootstrap, the application looked much more modern and refreshed. The updated app garnered much attention and even had an article on New York Times dedicated to it.

With so much that has been built on top of the Velocity stack, migrating this monolithic and complex code is quite a daunting task. Quickly we discovered the hard part of the migration is not writing new code but understanding the legacy code to ensure we are migrating all the moving pieces correctly. This task became one of the most time-consuming tasks as the validation in the legacy Velocity was separated into configuration files, html files as well as Java files. Another challenge is that we are touching a live system where millions of dollars are spent daily. Any system error could be quite costly. We added integration tests and we focused on negative and error cases to ensure that an incorrect configuration would not result. We added tons of unit tests to ensure our front end code is bug-free. Introducing too many libraries across different product areas that seemingly performed the same tasks made our migrated platform fragmented. Since then, we have unified libraries across the different product suites to ensure uniformity across Turn codebase. We also had mix of Bootstrap 2 and Bootstrap 3 in our codebase, however, we have reigned back and chose Bootstrap 2 as our application is more desktop-focused and mobile-first is not what we need.

Due to the large amount of data we had to render with our application (80 column and 100 row tables and complex graphs), Angular does suffer from performance issues. The two-way binding and Angular’s constant monitoring of changes on the page does not play well for large data sets. Many listeners and watchers trying to make updates will grind the application to an halt. The developer needs to be extra careful to make use of these built in Angular magics. Having a client-side rendered application is not without problems. Client-rendered application is also at the mercy of the browser and the computer’s hardware, which made the performance of our application inconsistent across different sets of browsers and computers.

We are constantly making changes to components to speed up the application. For example, we brought in React to solve the problem of rendering large data sets. This has worked well, and we will dive deeper into this on a separate blog. Although universal rendered application is one of the hottest topics today and will definitely make the Turn Platform more performant, migrating to that will be costly. What we will do going forward is optimize the slow performing components and make the most of what we have.

 

Application Data:

engineering 
transforming_the_web_user_interface 
path /srv/www/sites/turn-dev.com/dev/repo/build/app 
main_controller app\controllers\Primary 

Request Data:

$_GET
No Data
$_POST
No Data
$_COOKIE
No Data
$_FILES
No Data
$_SERVER
REDIRECT_STATUS 200 
HTTP_HOST turn.stage.elusive-concepts.com 
HTTP_ACCEPT_ENCODING x-gzip, gzip, deflate 
HTTP_USER_AGENT CCBot/2.0 (http://commoncrawl.org/faq/) 
HTTP_ACCEPT text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 
PATH REMOVED 
SERVER_SIGNATURE Apache/2.4.10 (Linux/SUSE) Server at turn.stage.elusive-concepts.com Port 80 
SERVER_SOFTWARE Apache/2.4.10 (Linux/SUSE) 
SERVER_NAME turn.stage.elusive-concepts.com 
SERVER_ADDR 192.168.1.201 
SERVER_PORT 80 
REMOTE_ADDR 54.225.54.120 
DOCUMENT_ROOT /srv/www/sites/turn-dev.com/prod/webroot 
REQUEST_SCHEME http 
CONTEXT_PREFIX  
CONTEXT_DOCUMENT_ROOT /srv/www/sites/turn-dev.com/prod/webroot 
SERVER_ADMIN roger.soucy@elusive-concepts.com 
SCRIPT_FILENAME /srv/www/sites/turn-dev.com/prod/webroot/index.php 
REMOTE_PORT 58790 
REDIRECT_URL /engineering/transforming-the-web-user-interface 
GATEWAY_INTERFACE CGI/1.1 
SERVER_PROTOCOL HTTP/1.0 
REQUEST_METHOD GET 
QUERY_STRING  
REQUEST_URI /engineering/transforming-the-web-user-interface 
SCRIPT_NAME /index.php 
PATH_INFO /engineering/transforming-the-web-user-interface 
PATH_TRANSLATED redirect:/index.php/engineering/transforming-the-web-user-interface/transforming-the-web-user-interface 
PHP_SELF /index.php/engineering/transforming-the-web-user-interface 
REQUEST_TIME_FLOAT 1505977179.721 
REQUEST_TIME 1505977179 

Logs:

Time Data
2017-09-21 06:59:39
Loading Framework...
2017-09-21 06:59:39
app\models\Slug::lookup: Content_Slug::lookup(): No record found!
2017-09-21 06:59:39
2017-09-21 06:59:39
2017-09-21 06:59:39
app\models\Slug::lookup: Content_Slug::lookup(): No record found!

Events:

Event Data Listeners
APPLICATION >> RUN null 0
APPLICATION >> LOADED null 0
APPLICATION >> HANDOFF null 0
TEMPLATE >> HTML_START "" 0
TEMPLATE >> BEFORE_HTML_END null 1

Errors:

Notice (8) Undefined variable: clean_author /srv/www/sites/turn-dev.com/dev/repo/build/app/controllers/class.engineering.php L: 305
Notice (8) Undefined index: f /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Trying to get property of non-object /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Undefined index: f /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Trying to get property of non-object /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Undefined index: f /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Trying to get property of non-object /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Undefined index: f /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Trying to get property of non-object /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Undefined index: f /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Trying to get property of non-object /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 72
Notice (8) Undefined index: image /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 137
Notice (8) Undefined index: facebooklink /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 161
Notice (8) Undefined index: twitterlink /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 165
Notice (8) Undefined index: linkedinlink /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 169
Notice (8) Undefined index: pTitle /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 175
Notice (8) Undefined index: excerpt /srv/www/sites/turn-dev.com/dev/repo/build/tmp/smarty/templates_c/f7a7186590639363f640fdf7c56578099adb66c0.file.post.tpl.php L: 177

Benchmarks:

Benchmark Tag Time Comment
execution_time TIMER_START 0.000ms Starting bootstrap...
execution_time TIMER_STOP 116.291ms Debug console render output...