A Joomla Platform Library to compile and compress Javascript and CSS files

1. Is this project mainly a project for the Joomla CMS or for the Joomla Platform?

This project is for the Joomla Platform.

2. What is the goal of your project? What problem does it attempt to solve?

Today, many of web applications have a large amount of JavaScript and CSS files. For the reason of increased number of web applications and their rich content interactions, developers expect their applications to increasingly make use of CSS and JavaScript. Because these files determine how the page will be displayed on client browser, rendering the page in the browser is delayed until CSS and JavaScript files are completely downloaded. So it will take more time to render a web page, as the number of JavaScript and CSS files increases.

Why it takes that much time? Server generates a page in milliseconds. So that shouldn’t be the problem. Problem seems to be a combination of two things. Firstly, the JavaScript and CSS files with a lots of comment blocks and unnecessary spaces, new lines and longer variable names, which are meant for the developers but not for the end users, will increase the file size and hence the download time will be increased. Secondly, the large number of separate JavaScript/CSS files for a page, which cause one extra round trip per file to the server and back, will increase the number of server requests. So as the number of server requests increases, the page’s load time will be increased. To develop web applications which are nice and snappy to use, developers need to optimize the size and the amount of JavaScript/CSS files and make sure end users are having the optimum experience.

The goal of this project is to come up with a library for Joomla Platform which will facilitate compressing and combining of JavaScript and CSS files before they are sent to the client browsers. So developers can keep their well structured, commented and easy to read files aside and give compressed files which are optimized for speed downloading, to be served to the end users. And even more they will be able to send a single combined JavaScript/CSS file to render the page instead of sending a number of separate files. So developers will be able to reduce the page download time as well as to save their valuable bandwidth. This will benefit users with faster page loads and webmasters with reduced bandwidth usage.

3. Please describe the proposed project in detail.

Project Scope

The scope of the project is to implement a library for Joomla platform with following functionality. It will contain classes to compress JavaScript/CSS files using an algorithm, which will compress the contents of files with securing the output of the code and also classes which will do combining several JavaScript/ CSS files to a single compressed file thus reducing the bandwidth usage of web application. Each class will provide several functions to achieve above mentioned goals.

Research

There are several compression tools available to compress javascript and css files such as JShrink, JsMinifier, JsMin, YUI compressor, mrclay-minify, UglifyJS and csstidy, etc. They are implemented differently and in various platforms such as php and javascript. Some of them have drawbacks such as string variables and keywords are not guaranteed to be the same after compression. And also some of them may crash when processing a large amount of data. Below I have summarized the best steps used to compress a file in above mentioned tools. Most of tools mentioned above have not been updated in several years. So there may be better ways to do it. I have implemented a library for Joomla Platform and for now, I am using these steps to compress files and I am looking forward for feedback from mentors and community members to find out better ways of doing it.

Compressing JavaScript files

As a summary, we can compress a JavaScript by removing all comments, unnecessary white-space and shortening all local variable names to a single character. We can achieve this using several steps.

1. Backup Strings – Remove all strings from the JavaScript and store to reinsert later.
2. Remove Comments – Remove all comment blocks from the script.
3. Rename local variables – Replace all local variable names with a single character.
4. Preserve Keywords – Before removing whitespaces, backup Keywords in script that requires whitespaces and replace them with identifiers. (Eg: var, return, function… need to be followed by a space)
5. Remove unnecessary white space – Remove unnecessary spaces including tabs and new lines.
6. Restore keywords – Replace which were backed up in 4th step, with identifiers.
7. Restore Strings – Restore all strings which were backed up in 1st step.

Source : http://www.dynamic-tools.net/toolbox/javascript_compressor/

Compressing CSS files

Compressing a CSS file can be done in several steps.

1. Remove Comments.
2. Remove unnecessary new lines and semi colons.
3. Remove 1st digit of floats which are <1 (Eg: 0.56 -> .56)
4. Remove units of measure from zero values. (Eg: margin:0px -> margin:0)
5. Replace "none" with 0. (Eg: border:none -> border:0)
6. Remove unnecessary white space – Including tabs and spaces surrounding {}:; characters.
7. Compress HTML color codes if possible (Eg: #FF0088 -> #F08).

Source : http://developer.yahoo.com/yui/compressor/css.html

And also CSSTidy tool has an extra compression step as shown below.

source : http://www.if-not-true-then-false.com/2009/css-compression-with-own-php-class-vs-csstidy/

Combining Files

Combining files will facilitate reducing bandwidth usage by transferring a minimum number of files by combining several javascript/css files in to a single .js or .css file. JavaScript/CSS files can be compressed before combining in to a single file (compressing a file after combining will make problems as the large file size will causes errors as a result of PHP memory limitations). So files will be very much optimized for a speedy download.

source : http://rakaz.nl/2006/12/make-your-pages-load-faster-by-combining-and-compressing-javascript-and-css-files.html

Implementation

I have already implemented a prototype library with several classes with a number of working functions. These classes are declared in a way to support class auto loading feature of Joomla Platform.

I have implemented the static function compressFile() inside JCompress class. It will take three arguments sourcefilepath, destination file path and options. Below is an example of using JCompress::compressFile() function.

This function will make a relevent JCompressMinifier object depending on the file type and will compress the code and write it to the destination file. I have implemented some steps mentioned above under Compressing CSS files, in the JCompressCssMinifier class and Removing comments step in JCompressJavascriptMinifier class. I have tested JCompress::compressFile() function with layout.css file in beez_20 template for joomla CMS 2.5. The file size was reduced by 10KBs (30KB -> 20KB) and the compressed css file still gives the same output. With the support of the mentor, I will be able to develop a more advanced algorithm to achieve a higher compression ratio.

I haven't yet implemented functions for the combiner classes. I have done a research on combining JavaScript/CSS files effectively and I am looking forward to implement several working functions as soon as possible. I will push my work on combiner classes to my git hub repository of joomla platform which can be found at the link mentioned below.

There are several other working functions implemented already in some classes mentioned above.The implementation of the library is available here, in my Joomla Platform repository on github. This is just a prototype library and can be changed upon mentor’s concern.

Basic Approach of using this library

( According to the feedback given by community members for my post in platform dev mailing list and reviews on this proposal, I have understood that this approach has several drawbacks. I have suggested a better approach according to their ideas under topic - better approach)

This describes how compression library can be used. Developers can compress a js/css using JCompress::compressFile() method and use url of compressed_file when using addScript() and addStyleSheet() methods on JDocument object. Below is an example of that.

class TestApp extends JApplicationWeb{

...

$original = JPATH_SITE.'/css/main.css';

$compressed = JPATH_SITE.'/css/main_compressed.css';

JCompress::compressFile($original, $compressed, array("overwirte" => true) );

$this->document->addStyleSheet($compressed_file);

...

}

Or they can use JCompress::compressCss() / JCompress::compressJs() with addStyleDeclaration() / addScriptDeclaration() methods of JDocument object if they use small stylesheets/javascripts and prefer not to write files. Below is an example of that.

...

$this->document->addStyleDeclaration(JCompress::compressCss($uncompressed_code));

....

Better improved Approach of using Library

According to the feedback given by community members for my post in platform dev mailing list and reviews on this proposal, I have understood that previous mentioned approach has several drawbacks. Compressing/combining javascripts/css automatically will benefit in many ways. So rather than using above mentioned approach I suggest following approach for using this library.

1. Automatically compress/combine assets added to JDocument via native methods such as addScript() and addStyleSheet() by invoking a global function. We can use a configuration setting in global config to enable or disable this functionality upon developers choice. We can think of achieving this by implementing some override mechanism like the assets will be compressed/combined and registered in JDocument object when registering via native methods. For an example we can expand addScript() with additional parameters to enable/disable compression like addScript(.., $compress = true). Or we can get all registered assets from the fields such as _scripts, _styleSheets of JDocument object upon execution and compress/combine them before they are sent to client browser. In that case, we can think of overriding the JDocumentRenderHead::render() method. I am doing a research on this and I will be able to figure out the better way of doing it with the support of mentors.

2. Some times, there are assets that are loaded with traditional html ways (using link/scripts tags) rather than using framework methods. An example can be found here in index.php file of beez_20 template for Joomla CMS. Implementing functionality to automatically compress/combine these assets requires more research on this. I am looking forward it. We can think of parsing html output when rendering a page and setting a global configuration to enable/disable. An alternative solution would be keeping previously mentioned basic approach also enabled since developer is avoiding native methods and using html ways, he/she will need to manually use this library with the above mentioned basic approach. But this needs, re-coding of files/extensions which are already implemented, which is not a good thing.

3. There are some special cases, where scripts are loaded at the end of the document. We need to consider about them also.

As a summary, this approach will be more suitable for using the library.Please note that these approaches may change with ideas of mentors. I am going to do researches for implementing this approach. But before application  deadline, I haven't got enough time to come up with an idea how exactly it can be done. I will push my work to my github repository of joomla platform as I progress.

Caching files

Caching of combined/compressed files will increase the performance of the web application. Since different pages use different assets this will be little bit difficult to implement. I am looking forward to mentors' idea towards this.

Integrating with CMS

We can integrate CMS back end configurations with the library to decide which files to compress/combine, etc. And also we can use a global configuration to enable or disable compressing/combining by default.

Benefits over Extensions

Being a platform library, this will facilitate compressing/combining as a core platform functionality rather than through a extension. This will benefit both CMS and Platform developers when building their own web applications and will be available by default.

Problems to face

PHP memory limitations will cause errors when processing large files. As a solution, we can process a part of the file at a time.

References

1. http://developer.yahoo.com/yui/compressor/  
2. https://github.com/cgutierrez/JsMinifier
3. https://developers.google.com/closure/compiler/
4. https://github.com/mishoo/UglifyJS
5. http://www.crockford.com/javascript/jsmin.html
6. http://coffeescript.org/
7. https://github.com/einars/js-beautify
8. https://developers.google.com/closure/compiler
9. http://www.dynamic-tools.net/toolbox/javascript_compressor/
10. http://www.if-not-true-then-false.com/2009/css-compression-with-own-php-class-vs-csstidy/

4. What will be the end product of your project at the close of the program?

End product of this project will be a complete JCompress library for joomla platform with tests , documentation and with examples on using JCompress library.

5. What is your plan for completing the project on time? For example, what are key milestones and what is your rough schedule for achieving them?

6. What skills would a good mentor for your project have?

Familiarity with Joomla

1. Please describe your experience as a Joomla developer or if you have not worked with Joomla before, describe your plans for learning Joomla development.

I have been using Joomla CMS for about 3 years. I have developed several websites using Joomla CMS [1]. I have followed the book "Mastering Joomla 1.5 Extension And Framework Development" as three months ago I faced a situation where I needed to develop a small custom extension to a private website for students of my institution which I developed using Joomla 1.7 which includes a forum, ebook library and a JEvents component. Up to that I was just using extensions provided by Joomla Extensions Directory to develop my web sites. I was asked to develop a small extension which can check printer credit balance of a student from the printer server. That's the only experience I have in developing extensions for Joomla.

I have also followed developer docs for the latest releases of Joomla. I think I have a fair understanding about how Joomla works. I am also referring to api docs so I can expand my knowledge in Joomla API.

[1] http://www.anoonahope.lk | http://www.horanamidcitylions.com

2. Please attach or provide links to the repositories for one or more examples of your work with Joomla. This can be existing code or code you write as part of this application.

I have implemented a prototype library to compress and combine Javascript/CSS for joomla-platform. It is available in my github repository.

https://github.com/kaviththiranga/joomla-platform/tree/JCompress/libraries/joomla/compress

3. Please describe your familiarity with the development infrastructure used by Joomla, including PHPUnit, Selenium, PHPCS and Git. If these are new to you what are your plans for learning to work with them? We hope you will use the application period to increase your familiarity with these.

I am familiar with PHPUnit and Git. But Selenium and PHP CS are new things for me. I am referring to online docs and manuals available for them and I will be able to get an enough knowledge about them before coding period begins.

Involvement with the Joomla community

1. How have you been involved in the Joomla! Community? For example, have you participated in the Joomla Bug Squad, development mailing lists, the joomla.org family of websites, Joomla events or in other ways?

I have been following  joomla-dev-general@googlegroups.com mailing list since I started to learn Joomla extension and framework development. Now I am also following specific developer mailing lists for CMS and platform.

2. Have you released any extensions or provided support or services (volunteer or commercial) to Joomla users outside of the official Joomla project?

Yes. I am a member of Tech Team ( A team of students who provides support to other students in technical issues they face when using labs, etc.) in my Institution and we are maintaining a private web site called Tech Support Web which is built with Joomla 1.7. I have done a workshop for my team mates about using and administrating Joomla CMS.

3. If you have not been involved with the Joomla community before how do you plan to become involved?

I have not been involved in bug fixing and I am hoping to involve in bug fixing. I am searching for bugs which fit in to my experience. I am already in to few bugs and I will try to fix them as soon as possible.

4. Besides working on your project, what are your plans for involvement with Joomla? Do you expect to remain involved with Joomla after the completion of your project? If so, how?

Yes of course. I am willing to engage in Joomla Development as much as possible. And also if I am not accepted and no any gsoc student is working on this idea, I will continue implementing this library with the support of community members.

Participation in the Google Summer of Code project with the Joomla! Project

1. A requirement for successful completion of the project is that students publish a minimum of three blog posts or articles on the joomla.org family of sites. Do you agree to complete this requirement? 

2. A requirement for successful completion of the project is that you be in contact with your mentor or project administrator at least once a week. Do you agree to complete this requirement?

Yes.

3. During the summer there will be meetings, webinars and other possible online events for students and mentors. Do you agree to make a commitment to attend as many of these as possible given your time zone?

Yes.

4. The Joomla project has a code of conduct (lhttp://www.joomla.org/about-joomla/the-project/code-of-conduct.html).  Do you agree to comply with the code of conduct?

Yes.

5. The Joomla project requires code contributors to complete the Joomla Contributor Agreement http://developer.joomla.org/contributor-agreements.html. Have you completed this agreement or do you commit to completing it prior to the commencement of the program?

Yes. I have signed the agreement.