Thursday, September 18, 2014

Rewrite Technologies

Our team has been hard at work rewriting the OpenStax CNX site.  A frequently asked question is "What tech are you using?". This post covers a high level overview of the new site.

Architecture


Rewrite is a Single-Page App which means most of the logic lives in a Javascript client. The Client accesses data via REST APIs that are written in Python.  All of our data is stored in a PostgresSQL database. The details of the separate components follow.

Webview

Webview is the Javascript Client.  The basis of the Client is Backbone.js and Bootstrap.  We use several other Javascript packages as well.
Most of the Javascript is written in Coffeescript and compiled to Javascript.  CSS is developed using Less.

Webview requests json from the Archive via REST APIs.  The json contains the HTML for the content and any metadata. The Client parses the json and displays it.

Editing is a separate view in Webview.  When the user selects to edit, the views are swapped out. The new editor is based on the open-source HTML5 editor Aloha.  We have added several plugins to Aloha that are textbook editing specific.  The development on the editor was done by our team and the OERPub team with OERPub doing the bulk of the work.

We are using Nginx as our web server.

Archive

Archive stores published content and handles search. Content retrieving and search are handled via APIs written in Python. When content is requested, the json is built via stored procedures in Postgres. The json is built using the json functions in Postgres.

Search is done with optimized queries.  We are caching subject and one word searches long term and all searches short term to improve performance.  When a user pages through search results, the cached result is used to populate the next page.

Archive will run on an WSGI compatible server.  We are currently using Waitress as our server.

Publishing

The publishing application integrates with the Archive database. It allows users and third-party applications to publish content to the Archive, where it can be read and distributed to the public. Publishing is built similar to Archive, but differs in many ways. Archive is a read-only content API. Publishing provides an additional set of APIs that handle the publishing workflow, which includes user interactions like license and role (e.g. author or translator) acceptance, as well as the triggering of export files. Users of the OpenStax CNX system will typically never directly interact with this application. Almost all of the backend business logic is handled within the publishing application.

Authoring

Unpublished content is stored in Authoring in Postgres.  Authoring also has APIs used by the Editor and the Workspace. When content is published, an EPUB is generated and passed from Authoring to Publishing.  The EPUB format was selected to pass information between components because it encapsulates all of the info needed.

The Workspace is a listing of all content a user has access to edit. Books and Pages can be created in the Workspace and they can be deleted as well.

OpenStax Accounts

Users are now stored in a shared accounts component.  This was developed so users could have the same account on all OpenStax sites. All CNX users have been migrated to the new Accounts. Accounts uses OAuth so users will also be able to log in using Google, Twitter and Facebook. CNX will no longer create CNX user accounts.  New users will need to use one of their existing OAuth accounts to log in.

Logging

We are currently logging information, errors and user interactions to Syslog.  Our goal is to load the logs into Graphite so we can visually see how our site is being used.

Transformation Services

Transformation Services generates Export files(PDF, EPUB, Zip, etc.) and imports content to the editor. The initial design uses the same import and export code from Legacy CNX inside of a messaging system wrapper.

The messaging using RabbitMQ and several messaging queues. The queues will give us persistence of the file generation requests.  The requests will be sent by Publishing after content has been added to Archive. Publishing will pass an EPUB to Transformation Services that contains all of the data needed to generate the files.  

OpenStax CNX is a deceptively complex site that has many moving parts.  Our goal with this architecture was to design a component based system that can be easily updated and tested without impacting all of the site. All of our code is on Github.

Many thanks to CNX team members Michael Mulich and Derek Kent for reviewing and contributing to this post.

Friday, August 22, 2014

Tech Behind Search Improvements

Over the summer, we have made several improvements to the new version of the site (Rewrite),. One of the biggest improvements was with search. Since we released Rewrite, the performance of search has been a concern.  We did little to improve the speed before the release because of time constraints. It was decided that we needed to revisit it this summer to get things in great shape for the Fall semester.

We implemented the following changes:
  • Cache single word and subject searches - we added Memcached to store some searches.  A cron job is reloading the subject searches into the cache on a regular basis.
  • Pagination on Search Result page - previously we were displaying all of the search results on the same page. This caused a long page render time which made the slow search even slower. We are now displaying 10 items on each page. Books are displayed first since most users are looking for books. When possible, we use the cached search results to return the next page.
  • Improved SQL performance - we tweaked the SQL used for the queries to optimize them.
All of these changes are no-brainers, but have vastly improved search.

Monday, July 14, 2014

OpenStax CNX Development Tools

Over the last couple of years, we have changed our internal development tools. We do our own version of Agile development and have found these tools best meet our needs.

For Sprint planning, we use Trello.  Our team members are in many locations, so having a web-based tool to outline our Sprints has been very important. We create cards for User Stories or issues and work from the boards for Sprint planning and working on Sprints.


Our code is stored in Github. We previously ran our own SVN server, but slowly migrated all of our code to Github. It is a great tool. Our workflow for using Github is

  • Each component has a separate Repository.
  • Each Repository has a Master branch.
  • Each Repository has a production branch that contains the code currently in production.  This allows us to continue working and merging to Master, but also be able to fix problems in production easily from the production branch,
  • Developers branch off Master and code the Trello card they are working on. Once the code is completed and unit tested, the developer creates a Pull Request in Github. The Pull Request is to merge the code into Master.
  • A Pull Request triggers a code review by another team member.  Code reviews generally result in a review of the code as well as a manual test of the code.
  • Pull Requests are also unit tested using automated testing via Travis-CI
  • Once a Pull Request is approved, it is merged and the branch is deleted from Github.

Most of our meetings are held on Skype.  Skype has the simplicity of making a phone call and is mostly reliable.  We also use Google Hangouts when we need to share code or other screen sharing. It works really well, but if not as easy to start up as a Skype call.

Our team relies on IM. We have a Jabber server that some of the team uses and others use Google Talk or Hangouts.  IM is our virtual hallway and is a key part of our communication.

Wednesday, June 4, 2014

Publishing Added to OpenStax CNX Demo Site


We have recently added publishing to our Demo site.  If you are not aware, Demo is our alpha testing ground for the new version of our site,  Not everything is working yet and there are bugs remaining, but users can get a feel for what is coming soon to OpenStax CNX.

Users can

  • Create, edit and publish a new Page
  • Create, edit and publish a new Book 
  • Derive a Page or Book when viewing content
  • Derive a Page when editing a Book 
  • Publish a Book with published and unpublished Pages (the unpublished Pages get published with the Book). 
Please take a look and give us some feedback at techsupport at cnx dot org.  We would love to have you help us improve editing and publishing. The site will be updated regularly so check back often to see our progress. 

Tuesday, May 13, 2014

OpenStax College Android App Released

A new Android app that features the OpenStax College books was released on May 2nd.  The features include

  • Viewing released books
  • Saving Books or Pages to your Bookmarks
  • Take notes while reading
  • Export your notes to a text file or share them.
  • Share a Book or Page
The app is open source and the code is in Github. You can install the app from the Google Play store

The app can be used to access these books
  • College Physics
  • Introduction to Sociology
  • Biology
  • Concepts of Biology
  • Anatomy and Physiology
  • Introductory Statistics
  • Principles of Economics
  • Principles of Microeconomics
  • Principles of Macroeconomics


From a technical point of view, the app is based off of the existing OpenStax CNX Android app.  As part of our name change, the mobile version of our site was modified to work better with the Android app.  The result is a simplified look of the content (see screenshot).

If you use the app, we would love to hear feedback either as a rating of the app or by contacting us at android at cnx.org.

Wednesday, April 23, 2014

OpenStax CNX Featured On Floss Weekly Podcast


Two OpenStax team members were featured guests on the Floss Weekly podcast this morning.  Floss Weekly covers open source software for the TWIT podcast network.  Kathi Fletcher and Ross Reedstrom told the history of the project along with a discussion of the work we are currently doing on OpenStax CNX. The podcast is available to view or download from TWIT.

Thursday, April 3, 2014

New Name and New Editor Demo

In case you missed the announcement at the Connexions Conference this week, Connexions has been renamed to OpenStax CNX!  We have also renamed this blog to OpenStax CNX Developer Blog. You can visit us at openstaxcnx.org. The renaming give us a unique presence on the web and helps us build a family of OpenStax sites to provide open options for education.  Our other sites are available from openstax.org.


At the 2014 conference, we released a beta version of our new editor. It can be found at demo.cnx.org. Log in using your CNX, Facebook, Twitter or Google accounts and give the new editor a try.  We would love to hear your feedback as we continue to work on it.  It is definitely beta software so expect to find bugs.  We will continue to update it we get ready for the public release at the end of June.

This blog has been dormant for the past few months which is not good.  Our plan is to start posting regularly again since we have lots of cool code happening in the OpenStax College textbooks and in the rewrite of OpenStax CNX.