The PublicLab Rich Editor

The twelfth week marker! The beginning of the end! GSoC 2016 will soon come to an end. I am almost done with my work on the Q & A system though some work is still left as per my timeline goals. The only major work that remains is integrating the PublicLab Rich Editor for use in the Q & A system.

The PublicLab Rich Editor is a separate project in PublicLab that my mentor Jeffrey Warren has been working on. It is an Editor that supports both markdown and WYSIWYG content. It is to be integrated in the publiclab.org website for posting content very soon. As a part of Q & A project I thought of working on it a bit and contribute as much as possible. It wasn’t initially included in the timeline when I initially made my proposal but later I modified my timeline to include it as it really seemed interesting.

The  PublicLab Rich Editor is a s a general purpose, modular JavaScript/Bootstrap UI library for rich text posting, which provides an author-friendly, minimal, mobile/desktop (fluid) interface for creating blog-like content, designed for publicLab.org. It uses grunt for packaging and compilation. It has a rich text editor based on the Woofmark library and an autocomplete  feature  supported with horsey library. It uses jasmine as its testing framework.

Since I am not familiar with Nodejs or npm it is a bit tough for me to understand its modular structure. I still didn’t make any contribution to the Rich Editor. But I will likely make some contribution as the program wraps up. Have to learn Nodejs for this. I also have to work on fixes that come up during the last week as things are going to be deployed in the live site.

Also my team mates who were working along with me in the same website but different project are doing some awesome work and there is big merge on the Search project that is coming up. So things are getting busy in plots2 in the upcoming week. Stay tuned as the GSoC 2016 comes to an end!

Testing with Jasmine

This marks the seventh of GSoC 2016 and a lot of work has been merged since I started working. Looking back at it gives real feeling of accomplishment.

An important part of writing good code is that it is well tested. So that when you start working on a new feature or a fix you ensure that it doesn’t break any functionality by running the tests. Having a good test suite ensures your app has a strong backbone and it also can be extended for further development.

We have been writing rails tests for quite sometime and now we have good test suite for plots2 . But we didn’t have any test framework for testing JavaScript functionalities. So my mentor Jeffrey Warren suggested that we have tests for JavaScript too. We chose the Jasmine framework as it is the most popular JavaScript framework used and it also comes out in different packages for in browser testing and npm package as well as a rubygem as well for headless command line testing powered by phantomjs. And the best thing is it is best suited for Rubyists as the testing pattern is similar to Rspec – the popular Ruby testing framework. Here I am going to brief on how to setup Jasmine using the jasmine-rails gem. Since we were mostly using jQuery for JavaScript functionalities we also used the jasmine-jquery-rails gem to enable use of html-fixtures and some custom matchers.

Setup jasmine-rails

Follow the installation instructions to install the jasmine-rails gem to use it in your test-suite. This should also install the phantomjs gem along with it. But if the package installation fails you follow this documentation given here. A  spec/javascripts  folder will be created where all test files will reside.

You can run tests headlessly using the command

rake spec:javascript

 

How to write Tests ?

Now the most important how to actually write tests? The best place to start is with the jasmine docs. It gives a comprehensive guide to how to write tests. Jasmine tests basically describe a test and set some expectations which are tested against some matcher. All tests are written in spec files which are basically js files that reside in the spec/javascripts folder. Let us take a very simple test. As you can see it is written in a js file jasmine_spec.js


describe("A suite", function() {
it("contains spec with an expectation", function() {
expect(true).toBe(true);
});
});

view raw

jasmine_spec.js

hosted with ❤ by GitHub

Here the describe() function start a test description with a string and a function that contains a block of code. The it block also starts with a string and a function that contains a code block that contains expectations. The describe block basically tells the bigger story. And the it block tells the smaller individual stories that need to be tested. The expect() functions defines the expectation that is to be made. It can contain any instance or message. Basically any variable or method that returns something should go in it. The expect()function goes with a matcher chained with it. Each matcher implements a boolean comparison between the actual value and the expected value. It is responsible for reporting to Jasmine if the expectation is true or false. Jasmine will then pass or fail the spec. Here toBe() is a matcher.

Faking Ajax Calls

Now there are Ajax calls in applications that require server side requests. But depending on server responses makes the test suite slow and also leads to test failures when offline. So to test Ajax we need to fake Ajax calls using mock-ajax. You can simply copy paste the code in this link and put it in a helpers/mock-ajax.js file that goes inside the spec/javascripts folder. To mock ajax calls across a test suite you need to use install() and uninstall() in beforeEach()and afterEach() blocks in the beginning of the describe block as explained in the documentation. This sets up jasmine-ajax across the test suite. You can have a look at the jasmine-ajax documentation to have an idea on how to fake ajax calls.

Basically what happens in the entire process is you make or trigger a Ajax request first. Then you need to “Spy” on the Ajax call and return a fake Ajax response.  After that you need to test that with the expectation. A Spy is nothing but a function that helps faking the response. It is similar like a stunt double of an actor who fakes for the actor for a shot in a movie! You can learn more about Spies here. I was using jQuery and was using the .ajax() function for ajax calls so i preferred this stackoverflow answer to fake the ajax response.

Now you have been thinking what’s the point of testing if we actually setup the fake response  ourselves and test the same. Its definitely going to pass! Well  it’s not going to pass unless you have your Ajax call working properly. The entire point is not testing the response but whether the Ajax call is actually triggered on executing the action. The response is tested in the server framework, here, the Rails test suite.

Using HTML fixtures

When you are testing JavaScript in a web application you must have it associated with some html element may its a button or a link. What you want is to test it against some event like a hover or a button press specifically those associated with an Ajax call. What you need to do is to create a html file with that specific element like the button or link similar to what you have written in the actual page and load it in the test suite. HTML fixtures go in a separate fixtures directory inside the spec/javascripts directory. Here is a sample fixture I wrote .


<ul class="btn-group">
<li rel="tooltip" title="Helpful? Like it and get updates!" class="btn btn-default btn-sm btn-like" node-id="1" id="like-button-1"><span id="like-star-1" class="fa fa-star"></span> <span id="like-count-1">1</span></li>
</ul>
<script>
jQuery(document).ready(function() {
$('#like-button-1').on('click', clickliked);
})
</script>

view raw

unlike.html

hosted with ❤ by GitHub

As you can see I also associated the event associated with the button so that I can trigger the button press in the test.

To load the fixtures include the fixtures path at the top of the describe block

jasmine.getFixtures().fixturesPath="../../spec/javascripts/fixtures";

Also if you are using jasmine-jquey you need to preload the fixtures by including this line

preloadFixtures('unlike.html')

Finally load the fixture at the beginning of the specific test in the it block

loadFixtures('unlike.html');

Now you can use the html elements in the test. Like you can trigger the button click using

$('#like-button-1').trigger('click');

That’s how you use fixtures in test.

Whenever you have a problem in testing try debugging using a console.log() message. That helps you in figuring where you are going wrong in the test.

Here is how a spec file would look. You can find my work on this in plots2 PR #618.

Commenting System for Publiclab – Expanded Q & A Project

The fourth week of GSoC Coding Period is also over and the Mid-term evaluations are just about to come. So coming Mid-way of the GSoC journey, here are the final tasks that I completed before the Pre-midterm period.

  1. Modify existing comment system to handle answer comments and question comments
  2. Create new views and styles for comments in the Q & A pages
  3. Create expandable text boxes for comments and a View more functionality for comments

These are the brief overview of the features implemented. Let me now come to the technical details.

The current commenting system for Research notes uses a DrupalComment model for handling the comments in the database. So I used the same model for answer comments too. For answer comments I just had to introduce a aid column in the comments table and relate comments to answers. But I had to take care of a thing that answer comments weren’t related to the question otherwise all the answer comments would also show up in the question comments section in the page. So what I did was store a 0 in the nid field of answer comments to unrelate any answer comments to questions.

I also used the same comment controller for hadling requests for answer comments with some modifications. Took me to create new js partials for answer comments as comments were created and deleted witj Ajax requests.

The next work that I completed was introduce a new comments view for the Q & A pages. One of the new feature was creating a View more button for questions so that more comments were shown on pressing the View more button. Comments are listed in the descending order of their creation. So recent comments are listed at the top and older comments are shown as you Click View more.

Another cool feature that I implemented was expanding text boxes for comments. Did you ever notice how text boxes of facebook comments expand as you type in more lines without giving a annoying scrollbar. It’s the same feature that I implemented. I followed this article for that. I just used jQuery instead of raw JavaScript. It’s a really good feature when you want to save space as well give a good User Experience.

That’s all before the Mid-term evaluations. You can find this work in plots2 Pull request #589

Answering System for Publiclab – Expanded Q & A Project

Going over the third week in GSoC Coding period, a lot of Progress has been made over this week. A large chunk of Code has been merged and it makes me relieved that an important part of the Project is complete. This week I developed the Answering system for the Q & A Project which is the next most important part after the Questioning system.

Here is the summary of the tasks that I covered in the Answering system

  1. Create a new Answer model for handling Answers to each question
  2. Develop Post, edit and delete methods for Answers
  3. Design how Answers are listed for each Question in the Question page
  4. Create Like functionality for Answers
  5. Write functional and unit tests for the features implemented

These are the overview of the features implemented. Let me now go through the technical part of the implementations.

First is the Creation of the new Answer Model. As already mentioned in earlier posts all content in plots2 is handle by the DrupalNode model. Questions are a just type of node. Each question can have any answers so the Answer model is related to the DrupalNode model and also to a DrupalUser model that is the model which handles users in plots2 since an answer is written by an user. After setting up the relation, validations and various methods for the model had to be written for the model. Validations refer to the set of requirements that each record in the table must follow like the content of an answer can’t be blank. Rails defines a set of Validation methods for a model that handles this.

Next was creating a answers controller for handling the Post, edit and delete mechanisms for the answers. Answers are posted using Ajax requests in plots2. So is the delete action. Update action uses in-place edits for editing answers but the Update action is a normal http request. Working with Ajax in Rails is as simple as anything. Rails handles Ajax requests using non-obstrusive JavaScript. You just have to define a data-remote property in the links or forms for making Ajax requests and Rails will smartly handle each Ajax call. You can read more about it here.

Now coming to the Like functionality for questions. The Like button also involves Ajax requests. It is basically a button that Likes or unlikes an answer on button press. To implement the Like functionality a new AnswerSelection model had to be introduced. Eack like is related to the answer and the user who liked it. The answer_selection table has a liking field that takes a boolen value. It is set to true when an answer is liked and it is set to false if the answer is unliked. To deal with the Ajax calls I created a new answer_like controller which has a likes action that updates the liking field of the record.

Finally writing unit tests for the new models and functional tests for the new controller and implemented changes is a required part of good code.

On top of that I got to had a nice conversation with David Days and Ujitha of the Advanced search Team and moved the question Search functionality to a separate controller to avoid any conflicts and better collaborate with each other.

You can see the work on Answering functionality in plots2 Pull request #566.

Creating a Basic Search functionality

The second week of GSoC coding period is over and here are the works that I completed during this week.

This week was mostly spent on improving the design of the Questions page as proposed by my mentor and introducing a Basic question search functionality. I had to do some follow up fixes for my previous changes like adding comments for some temporary changes and one most important fix was to have a custom method defined for questions url so that it could be used at places required instead of writing out the full path every time.

Following the design proposed in plots2 issue #554 I modified the design for the questions page using some of the dashboard templates used in plots2.

You can now see the new questions page at https://publiclab.org/questions/ .

As proposed in the design I made a Basic Search functionality for Questions. Here I will walk through How I made the Basic Search functionality for questions.

There was already a search functionality that searched for notes, wikis, maps and comments from the  search box in the navbar. Autocomplete results are shown as you type in any keywords showing a icon and title in the autocomplete results suitably linked with the item. There is also an advanced search which you can see in the url  https://publiclab.org/search/ . There you can search selectively for any research note, wiki, map or comment by checking the checkboxes present there.

The question search is similar to the search except that it searches only questions. The actions for the questions search would go in the search controller. So the search query searches for any notes whose title matches the keyword and also has a question:foo tag. The search query for that is


@notes = DrupalNode.where(
'type = "note" AND node.status = 1 AND title LIKE ?',
"%" + params[:id] + "%"
)
.joins(:drupal_tag)
.where('term_data.name LIKE ?', 'question:%')
.order('node.nid DESC')
.page(params[:page])

view raw

search_query.rb

hosted with ❤ by GitHub

Here the :id is the search keyword typed. You can see my previous post for the definition of nodes and tags used in the codebase.

Now that I have fetched the questions I have to implement the autocomplete feature. I did it using the jQuery typeahead function, similar to what was done before for the search feature.

The JavaScript code for typeahead goes as follows


$('#questions_searchform_input').typeahead({
items: 15,
minLength: 3,
source: function (query, process) {
return $.post('/questions_search/typeahead/' + query, {}, function (data) {
return process(data);
})
},
updater: function(item) {
var url;
if ($(item)[0] != undefined) url = $(item)[0].attributes['data-url'].value;
else url = '/questions_search/' + $('#questions_searchform_input').val();
window.location = url;
}
})

I have a input field with an id ‘questions_searchform_input’. The items option tells the maximum items that can be listed in the results. The minLength option specifies the minimum characters that will trigger the search. The source option gives the dataset of search results. Here we have used a ajax post request that submits the keyword to the /search/questions_typeahead/:id  url.

The actual work of fetching the results is done in the search controller. It has two actions questions and questions_typeahead. The questions action shows the results searched by the keyword and displays them on submit. If no match is found it redirects the user to the post form to post  new question. The questions_typeahead action generates the results for the autocomplete results.

You can find my work on the search functionality in the commit ceabfb0.

Lastly I wrote some functional tests for the search controller and also an integration test to test the entire search mechanism. You can find my full work in the plots2 Pull Request #555

This is just a simple search functionality. There is much more to it for developing a advanced search functionality. In fact there is a parallel Advanced Search Project going on in Publiclab lead by David Days. In course of time the search code will be modified to meet the advanced search needs.

 

Questioning System for Publiclab – Expanded Q & A Project

The GSoC Coding Period has already started and its been a week since then. This post contains the work that I have done during the first week of Coding period.

Here are the list of tasks that I worked on during the first week

  1. Create a questions page that lists all the questions asked in the community.
  2. Create sections in questions page that shows Recently viewed, Most viewed and Most liked questions. They are shown in Recent, Popular and Liked tab respectively.
  3. Create a questions show page that displays the question along with its description.
  4. Placing buttons for editing, deleting and liking the question in the show page.
  5. Modify post mechanism for questions.
  6. Writing unit tests for the implemented changes.

Coming to the technical point of the implementations let me just walk through the Basic structure of the resources used for posting Research Notes in Publiclab.

All Research notes and wiki pages in plots2 codebase  are stored as nodes and accessed by the drupal_node model (Don’t think this has something to do with drupal. They probably migrated from drupal to Rails hence the name). In the current codebase a node has basically three ‘types‘. It is either note, wiki or maps.

plots2 has a tagging system where tag names are stored in a term_data table accessed by the drupal_tag model. There are some tags called power_tags that have a name like something:foo.

Questions are notes that have a question power_tag i.e. the tagname is question:foo. So for listing questions in the question page all I had to do was to list the notes with such tags. I had to create a questions controller for this purpose. The Recent tab listed questions in the descending order of post creation date. The Popular tab listed questions in descending order of the number of views. The Likes tab listed questions in the descending order of the number of likes.

Since there was already an posting mechanism and questions were actually a special kind of notes I had to just reuse the code and eliminate any redundancy. Any piece of software you write must always take care of this. Rails already supports this by writing DRY(Don’t Repeat Yourself) Code. So for post, edit and delete actions I had to modify the notes controller it so that it could work well for questions. Since the work is in the development phase I also had to take care that users don’t accidentally land up on these pages while they try to post a question or edit it that have a question:foo tag. This was taken care off by appending a :redirect parameter in the url of post and edit paths for the new feature. The post request would land up on the questions show page only if there is :redirect parameter in the url set to question.

Another bug that I solved in the way was that previously pages that required logged in access didn’t redirect users to the specific page rather they were redirected to the user dashboard page. I solved it to make it running. This could be used for the Ask a question button that required log in. Once the users are logged in they were taken to the post form.

Finally I had to write thorough test code for all the features implemented. Any codebase that needs to be used in long run must have thorough testing. My mentor Jeff  also supported this fact. Though I had experience of writing tests in Rspec, a Ruby testing framework the plots2 repository used Rails test::unit for testing purposes. So I started learning how to write tests in test::unit. It is really good to see the green dots for passing tests that you write! Once you get a hang of it it’s really fun writing test code! The Rails testing guide has some really good documentation on writing tests.

You can see my full work on this in plots2 Pull Request #550.

Parsing RSS feeds

There are many ways of Parsing RSS feeds data so that it can be used for showing useful information in a html format. Here I will show how we parse rss feed data using JavaScript that can be presented  in a webpage.

RSS is a type of document that uses standard web feed format for publishing frequently updated data like blog posts, headlines, audio or video. It is analogous to xml which just documents data primarily rather than rendering them. RSS feeds have a standard format of representation. You can find the specifications in this link. Here is a sample RSS 2.0 feed that documents map data for an author in mapknitter.org maintained by Publiclab.

To parse the rss feeds I used the jQuery.get() method which is function that handles Ajax requests. We use this method to load the rss feed using a get request


$.get('https://mapknitter.org/feeds/author/waren&#39;, function (feed) {
// processing code goes here
})

view raw

jQuery_get.js

hosted with ❤ by GitHub

As you can see in the RSS feed each item is surrounded by an <item> tag and the whole content is wrapped by a <channel> tag. The $.get() function on Ajax success returns the response in the feed variable. The feed variable thus has the content of the RSS feed.

Suppose we want to get the title and description of items in the feed. The processing code will then be


$.each($(feed).find('channel item'), function (i, item) {
title = $(item).find('title').html(),
description = $(item).find('description').html();
// Rendering code goes here
})

Similarly, we can get other elements of the item. Now we can use this data along with some JavaScript to render it on a webpage.

I had a table within a div with an id ‘maps’ to list all maps fetched from the rss feed data. So what I did was append some rows and cells to it to list them . The rendering code was similar to this


$('#maps table').append('<tr class="feed-item-' + i '"></tr>');
var itemEl = $('#maps table .feed-item-' + i);
itemEl.append('<tr class="title"></tr>');
itemEl.find('.title').html(title);
itemEl.append('<tr class="description"></tr>');
itemEl.find('.description').html(description);

So  you see it’s pretty simple Parsing RSS feeds using some JavaScript code. You can see my actual work in commit 4127568.