Huy's blag

Using some on-line resources (for links see below), I managed to deploy bleeding edge Typo on Heroku. Despite having a virtual root server to run Typo on, I decided to have it run somewhere else, because of the limitations on my virtual root server. Namely the number of open files, for Typo wouldn't have been the only thing running on it.

Preparations

Getting Typo is not exactly difficult. All you need is git and it's just

$ git clone git://github.com/fdv/typo.git
$ cd typo

to download the most recent version of Typo into a directory typo and jumping right into that directory. And since we are working with Heroku, installing the Heroku gem provides a useful command-line interface to your Heroku app:

$ gem install heroku

Setting up the repository for Heroku

Before you can do a git push to Heroku, you need to do a few things first. They all deal with the fact that the Heroku file system is read-only by creating empty folders (and keeping them in git) and disabling write access wherever needed.

Empty folders

Empty folders first. As git ignores empty folders, we need empty files within those (then not-so-empty) folders get them into git.

$ mkdir -p public/files tmp/cache
$ touch public/files/.gitkeep tmp/cache/.gitkeep
$ git add -f public/files/.gitkeep tmp/cache/.gitkeep
$ git commit -m "Adding empty directories for Heroku."

Removing write access

Now we need to disable write access in whatever might be writing to the file system. Create a file config/preinitializer.rb containing

require 'fileutils'

file_utils_no_write = FileUtils::NoWrite
Object.send :remove_const, :FileUtils
FileUtils = file_utils_no_write

to disable copying assets from the plugins to the public folder. This seems safe to do, because Typo comes with those files in the right place already.

I haven't tried using default themes, but using my own one, I've run into HTTP 500 errors when trying to get files like /images/theme/foobar.jpg. This presumably happens because caching fails (due to read-only restrictions), so I disabled this behaviour in config/environments/production.rb:

--- a/config/environments/production.rb
+++ b/config/environments/production.rb
@@ -7,7 +7,7 @@ TypoBlog::Application.configure do
 
   # Full error reports are disabled and caching is turned on
   config.consider_all_requests_local = false
-  config.action_controller.perform_caching             = true
+  config.action_controller.perform_caching             = false
 
   # See everything in the log (default is :info)
   # config.log_level = :debug

This works for me, but it seems to slow down the site by quite a bit. So maybe try and store images, stylesheets, and JavaScript files somewhere else and link to them.

Lastly, the CKeditor tries to write stuff to public/javascripts. Fortunately, StackOverflow is a good friend and knows a few solutions to that problem. I went with this editing vendor/plugins/easy-ckeditor/init.rb:

--- a/vendor/plugins/easy-ckeditor/init.rb
+++ b/vendor/plugins/easy-ckeditor/init.rb
@@ -1,9 +1,9 @@
 # Include hook code here
 require 'ckeditor'
 require 'ckeditor_version'
-require 'ckeditor_file_utils'
+# require 'ckeditor_file_utils'
 
-CkeditorFileUtils.check_and_install
+# CkeditorFileUtils.check_and_install
 
 # make plugin controller available to app
 config.autoload_paths += %W(#{Ckeditor::PLUGIN_CONTROLLER_PATH} #{Ckeditor::PLUGIN_HELPER_PATH})

As pointed out on StackOverflow, a pretty ghetto-ish solution...

$ git add config/preinitializer.rb config/environments/production.rb vendor/plugins/easy-ckeditor/init.rb
$ git commit -m "Disabling write access where needed for Heroku."

Databases and gems

The easiest way to let Heroku know which gems to install probably is providing a Gemfile.lock file. Doing this instead of creating a .gems file manually will ensure that all required gems are being listed, which may be relevant if the Typo team decides to add a gem for some new feature (or just use more recent versions of some gems).

Before we can get a Gemfile.lock file, we need to set up the config/database.yml file containing this:

development:
  adapter: sqlite3
  database: db/development.sqlite3
  pool: 5
  timeout: 5000

test:
  adapter: sqlite3
  database: db/test.sqlite3
  pool: 5
  timeout: 5000

production:
  adapter: sqlite3
  database: db/production.sqlite3
  pool: 5
  timeout: 5000

With config/database.yml in its place, we finally can fire bundle to get the Gemfile.lock we need.

$ bundle
$ git add -f Gemfile.lock config/database.yml
$ git commit -m "Adding Gemfile.lock and database.yml for Heroku"

If bundle fails for you, install the bundler gem first:

$ gem install bundler

This step is a major detour from what I have found on this topic. Chances are I'm doing too much here by implying the need of sqlite3 or so. But I think this is the more robust way to provide information about which gems we need to Heroku.

reCaptcha

For a while now (I don't know since when, as I had patched Typo for reCaptcha support some time ago), the master branch of Typo also supports reCaptcha. In order to use it, you need to enable it in the administration interface and have your reCaptcha keys, which you can get on the reCaptcha website, stored in config/initializers/recaptcha.rb.

$ git add config/initializers/recaptcha.rb
$ git commit -m "Adding reCaptcha keys."

Ship it to Heroku!

Since deploying to Heroku is as easy as a git push, that's all there is.

$ git push <your-heroku-git-reposiory> master:master

The last step to your Heroku-hosted Typo blog then is setting up the database.

$ heroku rake db:migrate

Congratulations on your own Typo blog on Heroku!

Custom Heroku domains with lighttpd

If you want your Heroku application to be reachable at http://yourdomain.com, there are a few possibilities. Heroku suggests setting the A record of your domain or subdomain to Heroku IP addresses. This may or may not be a viable option to you, but it was not one to me.

Fortunately, I found another way that did not require me to mess with the A record of my domains: Using a lighttpd reverse proxy to forward HTTP requests to Heroku:

$HTTP["host"] =~ "^(huydinh|informagics)\.eu$" {
  proxy.server  = ( "" =>
    (
      ( "host" => "75.101.163.44" ),
      ( "host" => "75.101.145.87" ),
      ( "host" => "174.129.212.2" ),
    ),
  )
}

This only sends HTTP(S) requests to my domains to the Heroku servers, while leaving all other requests untouched. A way more preferable solution to me.

To do...

As I had mentioned before, you maybe want your stylesheet/image/JavaScript files to be served statically instead of having Typo deal with /images/theme/* types of files. If you have a lighttpd (or any other web server) available anyway, it might as well offer such static files. I haven't implemented it yet, but chances are I will do so soon enough. Depending on what NewRelic and friends tell me.

So, this year's EuRuKo is over and next year's EuRuKo will be in Amsterdam. I guess that means I will have to camp them tickets even harder...

Party! Not...

Unfortunately, I did not have time for either of the three "parties" at c-base, Tante Käthe and somewhere else each. For that I blame... "someone". ;-D

From what I've heard, however, getting to Tante Käthe was more difficult than the organizers have hoped it'd be. Not only was it pretty far from the conference site, but it was very well hidden, too. Getting there in the dark did not help much either. I hope Amsterdam will feature some more easily reached party site.

Next year, if I am going to be in Amsterdam, I will have to make up for this year. Promise.

Talks

Most insightful to me were both Paolo Perrotta's talk about method_missing() and Jose Valim's talk about writing programming languages. Please don't get me wrong, I don't mean to say all other talks were not interesting. For example, Mateusz Drozdzynski's talk about sentiment analysis was great, as well, but not inducing a WHOA! as strongly as Paolo or Jose did to me, though.

While Paolo's talk probably benefitted a lot from Paolo's really strong performance on stage (not saying there was any lack of content!), Jose introduced me to a topic I haven't ever really got in touch with before. Plus his ideas were rather interesting, as well.

Some other talks however were... not that good. Some were just poorly presented and some had a severe lack of relevant content. But that's to be expected of conferences.

Ruby, Ruby, Ruby!

This. 'nuff said.

During the first four weeks of March 2011, 6 students in two groups each (totaling 12 students) had to implement an application for Android devices. The application was supposed to be able to record information relevant to the OpenStreetMap project, similar to what OSMtracker is capable of, technically. This project was particularly interesting, because the lecturer decided to give agile software development a chance. He never tried it himself, though. That's how interesting projects start!

Scrum!

I will not delve too much into Scrum here, because said four weeks were the first four weeks of Scrum of my life. It however was a really good decision of the lecturer to go the agile way, for if we had gone the "traditional" way, we probably would have been lost after no more than two weeks. None of us had knowledge on how to develop applications on the Android platform, which also meant that there were quite a few pitfalls out there waiting for us.

There also were a few changes to the requirements halfway through the project, which we could easily adapt to, thanks to agile processes. Such changes would most likely not have been viable if we had been "traditional". Getting feedback on progress for both the developers and the product manager also helped a lot when the eventual goal was not really that clear. After all, "The application is supposed to be able to record information relevant to the OpenStreetMap project" actually is not even remotely concise.

Implementation

Our group did make some very decent progress during the four weeks we had. In fact, it were not even four weeks, but just 16 days, because work times were from Monday to Thursday only, so people could work for their living on the other days. We have run into some of the aforementioned pitfalls, but software development on Android in general seems to be not too bad, as long as your Java is on a semi-decent level. It also helps a lot that most questions you might have have been posted on sites like StackOverflow already, so you usually will not have to spend days trying to solve a single problem.

One of the requirements of the product manager was that the application should be available for as many devices as possible, which essentially means that it should run on Android 1.5 (API level 3) and higher. We settled on Android 1.6 (API level 4) and higher, because it made life a lot easier when it was about the GUI.

The most difficult issue we had to solve probably was recording media files. Taking pictures was as easy as calling an intent for MediaStore.ACTION_IMAGE_CAPTURE, but the intent for video recording (MediaStore.ACTION_VIDEO_CAPTURE) did not work properly on some devices and did not work at all for some other ones. We had to resort to implementing our own video recording activity using the Android MediaRecorder class, but we could not really make it work for all devices either. At first, everything worked on all HTC devices running stock HTC Sense ROMs, but it would not work on a Samsung Galaxy S, a Motorola Milestone 2 or my HTC Desire Z running GingerVillain 1.5. Then, it did work on all HTC devices (including mine), but

Camera.Parameters p = camera.getParameters();
p.setPreviewSize(320, 240);
p.setPreviewFormat(PixelFormat.JPEG);
camera.setParameters(p);

would throw exceptions on the Samsung Galaxy S and the Motorola Milestone 2 still. We ended up not using the Camera class at all, which worked for all devices. Mine does not show anything in the preview surface, until recording actually has been started. ;-(

But apart from that, no really big implementation problems occurred. MapsForge, the map display library we used, would throw exceptions every now and then, but thanks to their maintainer and his patches, many of them have been fixed by now.

About non-implementation things

"Real" documentation (wiki pages etc.) was neglected for most of the time. We had Javadoc comments all over the place, but many of them did not adhere to common Javadoc standards. In fact, during the first week, coding style did not adhere to those, either. But thanks to Checkstyle, bane of all slackers and the auto-formatting functionality Eclipse offers, we got to a common basis on both code style and comment style pretty fast. We did use a Checkstyle configuration that was not as strict as the default one, though.

Another useful tool most definitely was FindBugs. It scans your code for common traps (as well as errors) and shows them with a short description about why something is bad. One trivial error it would unveil is:

public boolean isTracking() {
    // Should return private variable isTracking instead
    return isTracking();
}

Such tools, when used frequently and consequently, do improve code quality by a lot. It also makes it much easier to enforce the same code conventions, as well as comment conventions. When available for the programming language of your choice, you should make use of them!

My role in the project

Apart from being the Checkstyle sheriff, spelling/grammar police officer and translation idiot, I was mostly responsible for data acquisition, both within the application (media files) and outside (e. g. parsing miscellaneous information about OpenStreetMap tags), and other smaller tasks.

Recording media files had to be done in Java, which I haven't ever really learnt before, to be honest, but for parsing the OpenStreetMap wiki, using Java did not really seem intuitive to me. So I used Ruby instead, which, to my surprise, the lecturer did not object to. At least not explicitly. Seeing how he was condoning Ruby (better: non-Java), I wrote a little Ruby script using the geokit gem to send fake GPS data to the Android emulator.

Taking the lead

We slowly, but steadily took the lead in the race against the other group of six during the four weeks. I do not know of the problems they might have encountered -- there might have been some different ones, because they started to use the internal SQLite database very early on --, but being able to virtually walk around on the emulator freely (and very exactly, thanks to geokit, gave us a huge edge. Testing and debugging GPS-related things could be done much easier and much more efficiently that way, because you did not have to walk around outside.

As a nice side effect, it also allowed us to easily show such things on an emulator during mid-sprint presentation sessions. You can use the Android mock location feature to come up with an application to do the same thing on a real device, but writing the Ruby script for the emulator took me half an hour. With my Java skills (or rather the lack thereof), it most likely would have taken me half a day or more, even. I might implement such an application at some time, though...

Lessons learnt

The relevant things I have learnt during the four weeks are not explicitly related to what I have coded, but more about things like code style. Different people do have different opinions about such seemingly trivial things, and when everyone ends up doing it the way they want to, the result will be horribad. As such, I have appreciated Checkstyle, even though it was annoying and tedious at times. I will most certainly have a look at roodi, which is a similar tool for Ruby.

Eclipse also was an interesting thing, at least. I have used RubyMine, an IntelliJ derivate for Ruby, before, but I decided to try and get along with Eclipse, because everyone else in the team was using it and the benefits of having the same environment and IDE by far outweighed my personal preferences for IntelliJ. I'm glad to be back to RubyMine, though. ;-D

It was a fun project while it lasted, but I think none of us wants to see our application again anytime soon. If you however want to take a look at it, you can find it on Google Code.

Thanks a lot to my team. You made those four weeks fly really fast!

Weee, I got one of the 50 first tickets for EuRuKo 2011! I hope it's gonna be a lousy weekend, weather-wise, because it'd suck to be inside for what's like a whole two days otherwise. ;-D

See you there!