technical-fongmun

[technical-fongmun]

Server Monitoring

We’ve looked a few options of server monitoring and found that Sensu seems to be an appropriate one because its architecture is transparent and simple. Plus, it can be scaled to multiple machines. Sensu basically orders its followers to execute certain command lines to collect stats and send back to itself. Then, we can set Sensu to send that data to a visualisation tool. It’s simple (but not easy).

I want to write about how we setup server monitoring with Sensu on FreeBSD. We need:

Sensu (which includes sensu-server, sensu-client, and sensu-api)

RabbitMQ

Redis

Logstash

Elasticsearch

Kibana

Sensu suggests uchiwa; it only shows alerts. Visualising a metric over time (e.g. CPU usage over time) on Kibana is much nicer. Plus, we are already using Kibana for visualising other metrics (e.g. request latency)

To be continuted...

[technical-fongmun]

AngularJS’s editing form in ng-repeat problem

Imagine we have a list of items, each of which has an edit button. When we click on an edit button, the edit form for the item is shown. And when we click on another edit button, the current edit form is closed, and the new edit form for the new item is shown.

The code looks like this:

<ul> <li ng-repeat="item in items"> <form name="form.editingForm" ng-if="editingItemId == item.id"> ... display editing form ... </form> <div ng-id="editingItemId != item.id"> ... display item ... <button ng-click="setEditingItemId(item.id);">edit</button> </div> </li> </ul>

When I click on one edit button, I get the editing form. $scope.form.editingForm is a valid form instance. I am happy.

However, if afterward I click on the edit button of an earlier item, $scope.form.editingForm becomes invalid.

Please note that, if afterward I click on the edit button of a later item instead, $scope.form.editingForm is valid and correct

You can try the bug here: http://plnkr.co/edit/Fs0Ih47jlUJWSWLXPapr

What the heck is happening here? Why does the order matter?

I guess, at some point of time, there are 2 editing forms because AngularJS processes from top to bottom, and that makes $scope.form.editingForm becomes invalid.

The hacky solution to this is to set editingItem.id to null first, and schedule a timeout to set editingItem.id to item.id. For example:

$scope.setEditingItemId = (newId) -> $scope.editingItemId = null $timeout( => $scope.editingItemId = newId 0 )

With the above method, there'll be no 2 editing forms at any time

[technical-fongmun]

Scoping variables in ng-include without ng-repeat

First, if we use ng-repeat, then we don't need to scope the template. With ng-repeat, a local variable is for the template. For example:

<div ng-include="template.html" ng-repeat="item in items">

The variable item is created locally for template.html. In this case, there's no problem.

In our case, we would like to use the template multiple times without ng-repeat. Without ng-repeat, the template has access to the global scope. And that's a big problem

Let's assume our template looks like this:

<script type="text/ng-template" id="template.html"> {{name}}: {{age}} </script>

When we use the above template multiple times, we would like to have the ability to set name and age for each time.

The ideal usage looks like this:

<div ng-include-template="template.html" ng-include-variables="{ name: 'Tanin', age: 31 }"></div>

With the above usage, we can write our own directive that has its own local scope.

.directive( 'ngIncludeTemplate' () -> { templateUrl: (elem, attrs) -> attrs.ngIncludeTemplate restrict: 'A' scope: { 'ngIncludeVariables': '&' } link: (scope, elem, attrs) -> vars = scope.ngIncludeVariables() for key, value of vars scope[key] = value } )

And that'll work nicely.

ng-include is a little strange in a way that it doesn't allow us to define a local scope. So, with ng-include, a template is not that reusable.

The directive is an answer to several questions on StackOverflow. Here they are:

renaming a variable in ng-include

Angular passing scope to ng-include

How to specify model to a ngInclude directive in AngularJS?

ng-include on load with an object

Different ng-include's on the same page: how to send different variables to each?

AngularJS - How to render a partial with variables?

[technical-fongmun]

Query all ancestors in Postgresql

We have a tree structure stored in our Postgresql database. The table have these columns: id, name, parentId.

What we want to achieve is querying a record with all of its ancestors.

It turns out that it's quite easy with Postgresql. Here is the SQL:

WITH RECURSIVE "tree" AS ( SELECT * FROM "table" WHERE "id" = ? UNION ALL SELECT c.* FROM "table" c JOIN "tree" p on c.id = p.parent_id ) SELECT * FROM "tree";

And that's it.

[technical-fongmun]

Test Javascripts and Coffeescripts in playframework

We would like to test our Javascripts and Coffeescripts within Play. There’s the sbt plugin, sbt-jasmine-plugin. It uses the Jasmine framework.

There are 2 problems with sbt-jasmine-plugin:

1. The test cannot be written in Coffeescripts 2. It uses Mozilla's rhino and env-js. And env-js is not maintained anymore. Plus, env-js emulates a browser environment, which can be problematic. It's better to use PhantomJS.

We have built sbt-web-test-js to overcome the above problems.

sbt-web-test-js performs the below steps:

1. Utilises sbt-coffeescript for compiling Coffeescripts files to Javascripts files 2. Generates the Jasmine HTML file 3. Execute the HTML page with PhantomJS. 4. Parse the result from the HTML page

If you would like to test Javascripts/Coffeescripts in Play, please try sbt-web-test-js

[technical-fongmun]

Exclude directories from assets pipeline

Note: here's our Stackoverflow's answer on the problem

So, it is not simple to exclude certain files or directories from the assets pipeline. I have no friggin' idea why it is the case.

First, we need sbt-filter. It provides a filter as a pipeline stage. Then, we can put a FileFilter that only exclude the unwanted files/directories. The code looks like this:

includeFilter in filter := new FileFilter { def accept(pathname: File): Boolean = { pathname.getAbsolutePath.contains("public/uploads/") } } pipelineStages := Seq(filter, digest, gzip)

There is one notable thing.

We cannot exclude directories with the example provided in sbt-filter because a tring is implicitly converted to one of the NameFilter's subclasses, and they only look at file/directory name. Therefore, you can only exclude by name, and this also works:

includeFilter in filter := "uploads"

But if you have multiple directories named "uploads" and want to exclude only one of them, then it won't work anymore.

[technical-fongmun]

Adding separators into a list

There’s a sorted List[Long], and we would like to insert a separator between 2 elements where their number of digits differ. For example, there should be a separator between 99 and 100, but there shouldn’t be a separator between 98 and 99.

The easiest (not simplest) solution is to change List[Long] to a mutable list, iterate through it, and insert separator when needed.

However, we like a solution which doesn't require a mutable data structure. And here is one way to do it:

def separate(sortedLongs: List[Long]): List[Either[Long, String]] = { if (sortedLongs.isEmpty) { return Nil } Left(sortedLongs.head) +: sortedLongs.zip(sortedLongs.tail).flatMap { case (first, second) => if (first.toString.length != second.toString.length) { Seq(Right("-"), Left(second)) } else { Seq(Left(second)) } } }

[technical-fongmun]

Pivoting an array of arrays

We want to pivot an array of arrays:

> pivot(Seq(Seq(1, 2, 3), Seq(4, 5, 6), Seq(7, 8, 9))) Seq(Seq(1, 4, 7), Seq(2, 5, 6), Seq(3, 6, 9))

Please take 5 minutes to solve this problem. If you cannot solve it in time, just scroll down to see the answer.

.

.

.

.

Here's our solution:

def pivot(seqOfSeqs: Seq[Seq[Int]]): Seq[Seq[Int]] = { if (seqOfSeqs.head.isEmpty) { Nil } else { seqOfSeqs.map(_.head) +: pivot(seqOfSeqs.map(_.tail)) } }

Now the above solution assumes that all Seqs have the same length. What if all Seqs do not have the same legnth?

Let's define the requirement first. Maybe it should work like this:

> pivot(Seq(Seq(1), Seq(2, 3), Seq(4, 5, 6))) Seq(Seq(1, 2, 4), Seq(3, 5), Seq(6))

We can modify the above solution a bit to handle this:

def pivot(seqOfSeqs: Seq[Seq[Int]]): Seq[Seq[Int]] = { val filtered = seqOfSeqs.filter(_.nonEmpty) if (filtered.isEmpty) { Nil } else { filtered.map(_.head) +: pivot(filtered.map(_.tail)) } }

One can argue that the above approach is slower than using a Mutable structure. We agree. However, we value readability and simplicity over performance, unless performance becomes a problem.

[technical-fongmun]

ALTER COLUMN from TEXT to TEXT[]

Say, you have a text column and you would like to convert it to text[]. The question is how you fill the value of the new column.

The answer is we need to specify the transform expression with USING.

Here's how we turn the previous value into an array of one element:

ALTER TABLE "pantip_kratoos" ALTER COLUMN "content_groups" TYPE TEXT[] using array[content_groups];

Here's how we turn a text containing a JSON array into Postgres array:

ALTER TABLE "pantip_kratoos" ALTER COLUMN "image_urls" TYPE TEXT[] using regexp_replace(regexp_replace(image_urls, '^\[', '{'), '\]$', '}')::TEXT[];

For turning JSON array into Postgres array, I cannot find a less hacky way...

[technical-fongmun]

Test MultipartFormData in Play

Ok, so I’ve been trying to write tests for uploading file in Play. I was stuck with the fact that Play doesn’t offer a Writable class for turning MultipartFormData[TemporaryFile] into Array[Byte]. Therefore, I cannot create a FakeRequest of MultipartFormData[TemporaryFile].

I've found this Stackoverflow thread pointing to writeableOf_multiPartFormData.

But there are a few bugs in the code. So, I've fixed those bugs, and here's the working Writeable[AnyContentAsMultipartFormData]:

import java.io.File import play.api.http.{HeaderNames, Writeable} import play.api.libs.Files.TemporaryFile import play.api.mvc.MultipartFormData.FilePart import play.api.mvc.{AnyContentAsMultipartFormData, Codec, MultipartFormData} object MultipartFormDataWritable { val boundary = "--------ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890" def formatDataParts(data: Map[String, Seq[String]]) = { val dataParts = data.flatMap { case (key, values) => values.map { value => val name = s""""$key"""" s"--$boundary\r\n${HeaderNames.CONTENT_DISPOSITION}: form-data; name=$name\r\n\r\n$value\r\n" } }.mkString("") Codec.utf_8.encode(dataParts) } def filePartHeader(file: FilePart[TemporaryFile]) = { val name = s""""${file.key}"""" val filename = s""""${file.filename}"""" val contentType = file.contentType.map { ct => s"${HeaderNames.CONTENT_TYPE}: $ct\r\n" }.getOrElse("") Codec.utf_8.encode(s"--$boundary\r\n${HeaderNames.CONTENT_DISPOSITION}: form-data; name=$name; filename=$filename\r\n$contentType\r\n") } val singleton = Writeable[MultipartFormData[TemporaryFile]]( transform = { form: MultipartFormData[TemporaryFile] => formatDataParts(form.dataParts) ++ form.files.flatMap { file => val fileBytes = Files.readAllBytes(Paths.get(file.ref.file.getAbsolutePath)) filePartHeader(file) ++ fileBytes ++ Codec.utf_8.encode("\r\n") } ++ Codec.utf_8.encode(s"--$boundary--") }, contentType = Some(s"multipart/form-data; boundary=$boundary") ) } implicit val anyContentAsMultipartFormWritable: Writeable[AnyContentAsMultipartFormData] = { MultipartFormDataWritable.singleton.map(_.mdf) }

Here's how you can use it in test:

val req = FakeRequest(Helpers.POST, "/test") .withMultipartFormDataBody( MultipartFormData( dataParts = Map("test" -> Seq("testValue"), files = Seq( FilePart("someKey", "test.jpg", None, TemporaryFile(new File("/your_file.jpg"))) ), badParts = Seq.empty, missingFileParts = Seq.empty ) ) val res = route(req)(anyContentAsMultipartFormWritable).get

I hope this helps!

[technical-fongmun]

Frictionless deployment

Deploying a Play application is a challenge. Play has a built-in HTTP server, which is Netty (I think). Netty is very good and production-ready.

The main problem is that, when we deploy a new instance, we have to kill the old instance and start the new one. This means there are a few seconds window where the server is not running.

We want to solve that problem.

At Fongmun, we are using Nginx as a proxy server to help reduce loads on the Play server. For example, if an HTTP request is for an image, Nginx serves it directly without going through Play.

We have also found that Nginx can reload its configuration frictionlessly, which means it reloads without being down. That's the trick we use to achieve our frictionless deployment. Here are the steps:

1. Deploy and start the new Play instance with http.port=0. It will pick an available port automatically 2. Get the port from the PID with lsof 3. Create an Nginx configuration with the new port 4. Tell Nginx to reload its configs, and wait a few seconds 5. Kill the previous instance

We have orchestrated all of these steps through Capistrano. It's quite a dance.

[technical-fongmun]

Log a cron job

We write a crontab entry that looks like below:

37 13 * * * some_command >> /mnt/ebs/logs/cron_$(date +"\%F__\%T").log 2>&1

Please notice that we use $(date + "\%F__\%T"). The application directs its STDOUT and STDERR to the file cron_2013-05-14__13:10:20.log.

With this technique, it is easy to see the logs of a specific run.

[technical-fongmun]

Running a script within Play environment

We want a way to run a script within Play environment because we need to use the prod configuration for databases and etc.

It needs a bit of code because, for example, a database connector needs the Play app that is started.

package custom_script object CustomPlayScript { def main(args: Array[String]): Unit = { try { implicit val app = new DefaultApplication(new File("."), this.getClass.getClassLoader, None, mode) Play.start(app) val redis1 = new RedisPlugin(play.api.Play.current).sedisPool val redis2 = new RedisPlugin(app).sedisPool // // ... do some main ... // } finally { Play.stop() } } }

Please note that `redis1` and `redis2` are initialised from either `current` and `app`. `current` and `app` are the same thing.

And you can run the code below:

activator run-main -Dconfig.file=conf/application.conf custom_script.CustomPlayScript

Or in production, you can run:

java -Dconfig.file=conf/prod.conf custom_script.CustomPlayScript

[technical-fongmun]

Moving away from off-the-shelf Puppet modules

We have moved away from off-the-shelf Puppet modules. By off-the-shelf modules, we mean the module we can download from https://forge.puppetlabs.com/. Off-the-shelf Puppet modules are good if you would like to achieve tasks quickly. However, they are usually large and extremely complex (e.g. involves ruby code). There are some valid reasons for the complexity; they supports multiple Oses. They are highly configurable. 

One example is https://github.com/puppetlabs/puppetlabs-postgresql has more than 10 puppets files with several files of assets. We re-write it to contain only 1 file and support only what we need (e.g. FreeBSD, create DB, create roles). It looks like this:

include freebsd_pkg class postgresql( $package_name = 'postgresql94-server-9.4.1_1', $log_file_path = '/tmp/pgsql.log', $data_dir_path = '/usr/local/pgsql/data', $port = 5432 ) { freebsd_pkg { $package_name: } ~> augeas { $package_name: context => "/files/etc/rc.conf", changes => [ "set postgresql_enable YES", "set postgresql_data \"'${data_dir_path}'\"", "set postgresql_flags '\"-w -s -m fast -o \\\"-p ${port}\\\" -l ${log_file_path}\"'", ], } ~> exec { "service postgresql oneinitdb ${package_name}": onlyif => "test `ls -l ${data_dir_path} | wc -l` -eq 0", } ~> service { 'postgresql': ensure => true, enable => true, # For postgresql, 'enable' doesn't work. But for Redis, it works. WTF? } } define postgresql::exec_sql( $sql = $name, $onlyif = undef ) { include postgresql $psql = "psql -h localhost -p ${postgresql::port} -U pgsql -d postgres" if $onlyif { $onlyif_with_psql = "test `${psql} -tAc \"${onlyif}\" | wc -l` -eq 0" } else { $onlyif_with_psql = undef } exec { "${psql} -c \"${sql}\"": onlyif => $onlyif_with_psql } } define postgresql::db ( $database = $name, $user, $password ) { include postgresql postgresql::exec_sql { "CREATE USER \\\"${user}\\\" WITH PASSWORD '${password}';": onlyif => "SELECT 1 FROM pg_roles WHERE rolname='${user}'" } ~> postgresql::exec_sql { "ALTER ROLE \\\"${user}\\\" WITH PASSWORD '${password}';": } ~> postgresql::exec_sql { "CREATE DATABASE \\\"${database}\\\";": onlyif => "SELECT 1 FROM pg_database WHERE datname='${database}'" } ~> postgresql::exec_sql { "GRANT ALL PRIVILEGES ON DATABASE \\\"${database}\\\" to \\\"${user}\\\";": } ~> postgresql::exec_sql { "ALTER DATABASE \\\"${database}\\\" OWNER TO \\\"pgsql\\\";": } }

The above is much easier to maintain.

[technical-fongmun]

Puppet chicken-and-egg problem

At Fongmun, we use Puppet to manage our infrastructure, even though we have a single server. We want to resolve the question “Can we replicate everything on a new machine easily?”

When we get a new machine, we need to install Puppet before running our Puppet script. As a side note, we use Puppet Apply, not Puppet agent-master.

Therefore, we need to bootstrap the machine with Ruby, Rubygems (for installing Puppet), Puppet, and Git. When we have those, we can get our Puppet script from Git and run it.

Then, our Puppet script ensures that Ruby, Rubygems, and Git are installed.

Here is our bootstrap script on FreeBSD:

#!/usr/local/bin/bash if [[ `whoami` != "root" ]]; then echo "Please run the script as root. Try 'sudo su root'" exit 1 fi if [ ! -f /bin/bash ]; then ln -s /usr/local/bin/bash /bin/bash if [[ $? -ne 0 ]]; then echo "/bin/bash is not available. We tried symlink from /usr/local/bin/bash, but it didn't work. Please fix it." exit 1 fi fi portsnap fetch extract update if [[ $? -ne 0 ]]; then echo "Updating Ports collection failed" exit 1 fi pkg install --yes openssl-1.0.2_1 if [[ $? -ne 0 ]]; then echo "Installing openssl-1.0.2_1 failed. Please check earlier error messages." exit 1 fi pkg install --yes ruby-2.1.6,1 if [[ $? -ne 0 ]]; then echo "Installing ruby-2.1.6,1 failed. Please check earlier error messages." exit 1 fi pkg install --yes ruby21-gems-2.4.6 if [[ $? -ne 0 ]]; then echo "Intalling ruby21-gems-2.4.6 failed. Please check earlier error messages." exit 1 fi gem install puppet -v '4.0.0' if [[ $? -ne 0 ]]; then echo "Installing Puppet 4.0.0 failed. There might be some dependencies missing. Please check the earlier error message if there's one." exit 1 fi mkdir -p /etc/puppetlabs/puppet mkdir -p /etc/puppetlabs/code mkdir -p /opt/puppetlabs/puppet/cache mkdir -p /var/log/puppetlabs # Note: we cannot use pkg because ruby-augeas requires augeas-devel, and `pkg install augeas` doesn't install augeas-devel cd /usr/ports/textproc/augeas && make install clean BATCH=yes FORCE_PKG_REGISTER=1 && hash -r if [[ $? -ne 0 ]]; then echo "Installing Augeas 1.2.0.3 failed. There might be some dependencies missing. Please check the earlier error message if there's one." exit 1 fi gem install ruby-augeas -v '0.5.0' if [[ $? -ne 0 ]]; then echo "Installing ruby-augeas 0.5.0 failed. There might be some dependencies missing. Please check the earlier error message if there's one." exit 1 fi pkg install --yes git-2.3.7 if [[ $? -ne 0 ]]; then echo "Installing Git failed. You might want to check if /usr/ports/devel/git exists or check an earlier error message if there's one." exit 1 fi ssh -o StrictHostKeyChecking=no git@github.com if [[ $? -ne 1 ]]; then # yeah, it exits 1 if succeeding. With permission denied, it exits 255 echo "It cannot ssh into github.com. You might want to check if you forward SSH agent correctly." exit 1 fi cd / && git clone git@github.com:tanin47/fongmun.git if [[ $? -ne 0 ]]; then echo "Pulling github.com/tanin47/fongmun repo failed." exit 1 fi cd /fongmun/config_management && bash 'NOOP=false ./puppet.sh'

[technical-fongmun]

Back up data to Google Drive

Backing up and restoring data are essential but painful. We want to automate them, so they are simple and easy to perform. Ideally, the only human interaction required is pushing the button.

At Fongmun, we always aim to automate everything that can be automated, e.g. reading logs, backing up data, and restoring data. One good side effect of the principle is that automation artefacts serve as documentation. For example, we use the reading-logs script for reading logs and look at its source code to find out where the logs are.

We've chosen to backup our data to Google Drive because of the free 15GB space. Let's look at how we store our data on Google Drive.

Here's the overview of the backing-up task:

Copy the data file from Redis and dump the data from Postgres to multiple files

Zip all data files into a single ZIP file

Upload the zip file to Google Drive

Remove previous backed-up ZIP files that we don't need

The only complicated part in the above task is how to talk to Google Drive. We've learned a few things from building the script.

First of all, we upload a ZIP file using Ruby with the google-api-client gem from our server. The credential downloaded from the Google Developers Console is enough for consuming Google Drive API as the service account. So, our first lesson is:

Service accounts can consume Google Drive API without user action.

Service account can be created using Google Developers Console. You can create a project, go to "Credential", and create a new OAuth Client ID. And here comes the second lesson:

A service account is totally separated from its owning Google account. The service account has its own email address.

Since service account is separated, our third lesson is:

If the service account creates a file, its owning Google account won't be able to see it.

It isn't over yet. While trying to debug the script, we've discovered the fourth lesson, which is:

The service account's Google Drive cannot be accessed through web UI.

Be able to see the backed-up files is useful. We want to download them and inspect them. To overcome the inconvenience, we create a folder with the Google account and share it with the service account (using the service account's email). It allows us to see backed-up files through our Google account and upload backed-up files through our service account.

With the explained setting, we have a great and free backing up system!.

PS. The Ruby code for upload a file to Google Drive:

client = Google::APIClient.new( :application_name => 'Example Ruby application', :application_version => '1.0.0' ) key = Google::APIClient::KeyUtils.load_from_pkcs12('your_private_key_file.p12', 'notasecret') client.authorization = Google::APIClient::JWTAsserter.new( 'someaddress@gservice.com', 'https://www.googleapis.com/auth/drive', key ).authorize() client.authorization.fetch_access_token! drive = client.discovered_api('drive', 'v2') file = drive.files.insert.request_schema.new({ :title => 'test.txt', :mimeType => 'text/plain' }) client.execute({ :api_method => drive.files.insert, :body_object => file, :media => Google::APIClient::UploadIO.new('local_path_to_file', 'text/plain'), :parameters => { :uploadType => 'multipart', :alt => 'json' } }).data

[technical-fongmun]

Do NOT use multiple users for deployment with Capistrano

We’ve made a mistake of using multiple users within the same group for deployment with Capistrano. We thought it would be more awesome.

It doesn’t work well at all.

Mainly, because Capistrano creates a bunch of files under our own users and do not enable the write permission for the group on the recently created files.

That causes a bunch of issues with “Permission denied”. For example, it cannot remove old releases because some files in older releases belong to someone else (who did the deployment for those releases)

Just. Don’t. Use. Multiple. Users.