Knockout Asynchronous Computed Observables

With its Observables and Computed Observables, Knockout provides some simple mechanisms for binding data values. However, when it comes to capturing results of asynchronous calls in variables, things become a little bit more complicated. Here is what the difficulties are and what can be done about them:

Observables, computed observables and their limitations

An observable is a special kind of object that can have subscribers. If the value of an observable changes, its subscribers are automatically notified about that changes. This is how we can declare observables with Knockout:

[crayon-5d1340f3d40fb466392792/]

Now we can specify a function that computes a new value from these observables. This function is called a computed observable and its value is updated whenever any of the observables it depends on are changed:

[crayon-5d1340f3d40ff769713976/]

The problem is that both observables and computed observables have to return a value synchronously. However, sometimes we might need to do asynchronous calls such as AJAX requests and sometimes we might even want to use a computed observable to represent the data we fetch with this AJAX request.

Capturing asynchronous results in an observable

In order to still be able to catch asynchronous results in an observable, we can declare a further observable queryResults and use a computed observable to do the AJAX request. The information obtained by the AJAX request can then be stored in queryResults:

[crayon-5d1340f3d4100547075355/]

Having the AJAX request wrapped inside a computed observable will ensure that it is called not just once, but every time one of the query parameters (pageIndex and sortColumn) changes its value.

Even though this solution works, it is not very nice that it requires a separate observable queryResults; instead it would be more desirable to emit the output directly from the computed observable.

Asynchronous Computed Observables

A common way of handling asynchronous operations in JavaScript are Deferreds - objects that allow adding callback functions to callback queues so that you will be notified when the result becomes available. So if we can get our computed observable to return a Deferred object to represent the AJAX request, our problem will be solved.

In order to achieve this, we create a wrapper function asyncComputed around a computed observable. The computed observable captures the output of the deferred value and uses the callback to transfer the output onto the observable result, which is returned directly by the asyncComputed.

[crayon-5d1340f3d4101817134019/]

So if we want to make an AJAX call as reaction to an observable changing its value, we simply need to pass the AJAX call as evaluator to the asyncComputed like this:

[crayon-5d1340f3d4102868667733/]

As we can see, the asyncComputed can be used in place of a regular computed observable, and its result will appear asynchronously after any of its dependencies change. So we finally got the solution we have been looking for.


Integrating custom widgets in Wordpress using RequireJS and Almond

A couple of days ago I embarked on a quest to write my own Wordpress widget. The widget was to consist of a main file (widget.js), some libraries and another module encapsulating an algorithm I needed in the widget. That meant that I faced the task of including all that in my Wordpress page. This is how I solved it:

1. RequireJS

The easiest way to integrate my own widget with all its dependencies into the Wordpress project seemed to be via RequireJS. RequireJS is a JavaScript file and module loader which makes it unnecessary to reference every single required script file in the html document. Instead, the only reference needed is require.js. This results in the following script tag in my Wordpress page:
[crayon-5d1340f3d4307812466338/]
With the data-main attribute, I specify a single entry point for RequireJS. RequireJS will load the widget.js file first and check it for further dependencies. The dependencies are passed as an array of names to the module definition. Additionally, I use a configuration object (requirejs.config) to list any paths which are not found directly under the base url, so that my widget.js file looks like that:
[crayon-5d1340f3d430a867230218/]
Though this is a standard way to include modules in a project, it did not work for me. The problem was that the Wordpress project resides on one server, and the widget on another, and as it turned out, RequireJS could not cope with this cross domain request. So I needed to find a workaround.

2. RequireJS Optimizer

My next step was to use the RequireJS optimizer. The optimizer combines all JavaScript files and modules into one single file. Once this optimized file is created, we can reference it in the html. Since this newly created optimized.js file contains widget.js as well as any other libraries and modules required by the widget, we no longer need to load RequireJS into the html page. So I replaced the script tag from above by a new one:
[crayon-5d1340f3d430b422214896/]
In order to create the optimized.js file, we have two options: either attach all necessary arguments to the basic command line command, or create a build file which specifies the arguments. I opted for the latter and created a widget.build.js specifying the base url, the path of the output file, etc.

Now all I needed to do in the command line is pass the build file's name to the optimizer:

r.js.cmd -o widget.build.js

Using the RequireJS optimizer and referencing the optimized script file instead of require.js in the html's script tag solved the problem of the cross domain request, but with this problem solved, another arose: running the application in the browser, I got an error "define is not defined". Obviously, Wordpress was unfamiliar with the define() call, which was used at some place within the optimized.js file.

3. Almond.js

As a solution to this new problem, I decided to use Almond, an AMD API shim which replaces RequireJS. In order to use Almond, we use the RequireJS optimizer to create one single file, as we did before. The only difference is that now we pass the path to the almond.js file as name tag to the optimizer. Again, we can do that either in the command line or in a build file. Here is my final build file with the Almond reference:
[crayon-5d1340f3d430c413430770/]
Feeding these parameters to the RequireJS optimizer, I finally succeeded in loading my widget into the Wordpress page. Whew!


Getting Started with StrongLoop

What it StrongLoop

The StrongLoop API is a platform featuring the LoopBack framework - an open-source node framework which enables easy creation of REST APIs and the connection to backend data sources such as MySQL or MongoDB databases. In this Blog Post I will show how I got started with the framework and how I used it to created a simple web application.

Getting started with a small example app

For getting started with my first project, I followed the guide at http://strongloop.com/get-started/. Creating a StrongLoop project is easily done by calling the loopBack application generator with the command $ slc loopback. The generator will prompt you for a name and directory of your application and then generate a scaffolding StrongLoop project.

Setting up the StrongLoop project

The loopBack application generator is one of many slc command line tools, some of which we will encounter later on. Alternatively, StrongLoop also provides a graphical UI - called ARC - which can be used to execute the same tasks by button clicks, but personally, I was more comfortable with the command line. I liked that it leads you through the processes step by step by prompting you for one thing at a time.

1. Generating a scaffolding project

We will now have a closer look at the structure and components of the project we have just created:

File Structure of the StrongLoop Project

The project distinguishes between the client and the server parts of the application. We will find the client folder to be empty - this is where we will later place our application's front end files.
The server folder contains the default configurations, the default boot scripts for basic initialization (they will be run when the server starts) and the server.js wich will start the web server if we run the application. The database.json lists a default in-memory data source named db. I added a file name so that all the data we generate with the app (such as users, bookings, ...) will be written into that file:
[crayon-5d1340f3d4403913756877/]
Later on, we could add a real database here. For now, however, I want to focus on getting to know the basic ideas of StrongLoop, so a simple json file as data storage will be sufficient as a start.

Apart from that, our application already contains several built-in models (in the node_modules/common/models folder). Models represent backend data sources and always are created as a json file specifying the object parameters and an accompanying JavaScript file in which we can define further methods. The built-in models are models which are likely to be used in most applications, such as User, and thus save us time and effort to create them ourselves.

LoopBack also provides a predefined REST API for each model, comprising the CRUD operations (create, read, update, delete) - so again we are saved the trouble of creating this basic functionality by ourselves!

2. Experimenting with User REST Endpoints

Seeing that our application already has several models and REST endpoints, we can start exploring them in order to learn how they can be used. In order to do that, we need to start the server ($ slc run) and go to http://localhost:3000/explorer/, where we will find a list of the existing REST endpoints. By default, only the User model is exposed via the REST API, so we should not be surprised not to find the other built-in models there.The UI allows us to try out the different endpoints. We can, for example, click on POST /Users and create a new instance of a user by specifying a username, password, etc. The UI then displays the URL of the request we have just made, as well as its response code and response message. I found this "playground" very helpful for learning in what way to use these endpoints later in the actual application.

3. Adding custom models and custom REST APIs

So far, we have only talked about the application's default elements and functionalities. It is now time to expand the app by our own custom elements. For defining custom models, we can use the LoopBack model creator ($ slc loopback:model,) which takes us step by step through the setup of a model. Since I wanted my first StrongLoop app to be a small programme for making holiday reservations, I named my first model 'Booking' and chose fitting parameters:

Creating the Booking Model with the LoopBack model creator

This is the resulting booking.json which we find in the newly generated folder common/models. All custom models are placed into this folder automatically:

Resulting Booking Model

The booking.js file does not contain any code at this point apart from the declaration of the class. However, we also want to define our own custom REST Endpoints, so I added a remote method, which is a static method of a model that is exposed over the REST API. My method is called 'welcome' and simply creates a welcoming message. If we run the app again, we see that http://localhost:3000/explorer/ now also includes endpoints for our custom Booking model - the default CRUD operations as well as the custom endpoint GET /Bookings/welcome. If we try out the new endpoint, we get the following result:

Response to Bookings/Welcome

4. Integrating a client app

Having gotten the basic grip of StrongLoop and how to use and create REST Endpoints, I integrated my client app - an application for making holiday reservations. I use the REST Endpoints in my client app in order to login and register users, to retrieve information from the server and to create new reservations. The good thing about StrongLoop is that it leaves it up to the programmer to decide which languages and technologies to use for the client app, so you are free to chose to use Android, iOS or, as I did, the Angular.js framework.

What I liked about StrongLoop

I think that StrongLoop is a great framework for the creation of web applications because it is easy to learn and straightforward to use. It creates many useful default elements such as the user model, while at the same time maintaining a clear project structure. And I particularly like that you can view and try out each of the project's REST APIs, which is especially helpful if you are new to the subject of using REST APIs.

The next steps for me will be to delve into more advanced features such as user authentication and the inclusion of real databases.


Client-side form validation with AngularJS

Many web applications depend on forms to collect user input. However, as a rule, the user cannot be trusted to enter valid input values by himself, so we need a way to control which values the user is allowed to commit. A popular way to do so is via client-side form validation, since it can give immediate feedback about whether the form is valid or not.

With AngularJS, client-side form validation is really simple. The possibilities range from making use of the HTML5 input types and attributes, defining customized angular directives to showing custom error messages.

HTML5 form validation

In HTML5, the <input> tags provide different input types and attributes with which forms can be validated on the client side without having to write any JavaScript functions. These input types comprise types such as "number", but also more complex types such as "email", "date" or "url". If an input type is specified, the browser will not allow user input that does not correspond to that type.

If we want to make sure that the user cannot leave a field empty, we can add the 'required' attribute to the input tag. This way, the form cannot be submitted unless all input fields with the 'required' attribute are filled in and the browser will notify the user if required data is still missing. For additionally restricting the data format, we can use the 'min' and 'max' attributes, as is used for the age field in the following example:

[crayon-5d1340f3d4531503132136/]

AngularJS directive

If we use AngularJS for our web application, we are provided with further instruments to validate user forms. Some of the Angular directives can be used to specify the format of the user data:

[crayon-5d1340f3d4533380366570/]
On top of that, AngularJS allows us to define our own customized directives. We could, for example, create a directive that validates that an entered username is already contained in the database:
[crayon-5d1340f3d4534560714390/]

AngularJS Form Properties

Each AngularJS form has certain properties with which we can check the state of the form. These properties can be accessed via formName.inputFieldName.property. To be able to use these properties, all input fields must be given a name.
We can, for example, check the validity of a form with the boolean properties $valid or $invalid and enable or disable the submit button according to that value:

[crayon-5d1340f3d4535087871679/]

In order to communicate to the user if his or her data is invalid and explain which kind of input is expected instead, we can use the $error property to create customized error messages. With the help of the ng-show directive, we can arrange that the message is only shown when the error in question has actually occurred:

[crayon-5d1340f3d4536283725635/]

Because both the name and the age are required data, the corresponding error messages are displayed in the empty form:

empty form

After entering a name, the message 'name is required' disappears. If we try to enter an age which is outside the specified range, we get another error message:

age out of range

When we finally enter all the required data in the correct format, no error messages are displayed any longer and the 'Save' button is enabled:

valid form

Summary

We have learned about some handy tools which enable client-side form validation to be achieved without having to write JavaScript functions. However, we must not forget that even though client-side form validation is good for giving the user instant feedback and thus enhancing the user experience, it can still be bypassed by users with malicious intents. Therefore it is vital for a secure application to validate input data on the server side as well to prevent that potentially dangerous input is processed on the server side.


Anpassung des Soundex-Algorithmus für die deutsche Sprache

"Maier" oder "Mayer", "Schmidt" oder "Schmitt", "Hofmann" oder "Hoffmann"? Wer kennt das Problem nicht: eine Person ist in der Datenbank unauffindbar, weil die exakte Schreibung ihres Namens nicht bekannt ist.

Damit Namen auch bei ungenauer Schreibung gefunden werden können, darf der Suchalgorithmus nicht nur exakte Übereinstimmungen berücksichtigen. Eine solche unscharfe Suche (fuzzy search) liefert der Soundex-Algorithmus, der ursprünglich von Robert C. Russell für die Indizierung von Namen in einer Volkszählung entwickelt und im Jahr 1918 patentiert wurde. Dabei werden Wörter gemäß ihrem Klang so codiert, dass ähnlich lautende Wörter durch ähnliche Codes repräsentiert werden.

Das Prinzip des Soundex-Algorighmus

Die vom Soundex-Algorithmus gebildeten Codewörter bestehen aus einem Buchstaben - dem Anfangsbuchstaben des zu codierenden Wortes - gefolgt von drei Ziffern, die unterschiedliche Laute repräsentieren. Folgende Schritte werden bei der Codierung durchgeführt:

  1. Alle Buchstaben werden in Großbuchstaben umgewandelt und Satzzeichen (z.B. Bindestrich) aus dem Wort entfernt
    => 'Soundex-Code' wird zu SOUNDEXCODE
  2. Der erster Buchstabe des Wortes wird beibehalten
  3. Die Vokale "A", "E", "I", "O", "U", die sog. Halbvokale "W" und "Y" sowie das "H" (im Wortinneren häufig als stummer Konsonant gebraucht) sollen ignoriert werden und werden zunächst durch 0 ersetzt
    => aus SOUNDEXCODE wird S00ND0XC0D0
  4. Konsonanten werden gemäß folgender Tabelle durch Ziffern ersetzt
    Code Buchstabe
    1 B, P, F, V
    2 C, G, K, Q, X, S, Z, J
    3 D, T
    4 L
    5 M, N
    6 R

    => aus S00ND0XC0D0 wird S0053022030

  5. Alle Folgen von gleichen Ziffern werden durch nur eine Ziffer ersetzt; dadurch werden Doppelbuchstaben wie einfache Buchstaben behandelt, so dass "Hoffmann" und "Hofmann" und sogar "Hofman" als gleich betrachtet werden. Da zu diesem Zeitpunkt die Konsonanten bereits durch Codeziffern repräsentiert werden, werden außerdem auch Folgen von Buchstaben, die in die gleiche Kategorie eingeordnet wurden, als doppelte Buchstaben behandelt, so dass z.B. die Buchstabenfolge "ck" als ein Laut betrachtet wird.
    => aus S0053022030 wird S005302030
  6. Da die Vokale, H, W und Y ignoriert werden sollen, werden die Nullen entfernt.
    => aus S0053022030 wird S5323
  7. Das Codewort wird mit Nullen auf drei Ziffern aufgefüllt bzw. auf drei Ziffern verkürzt
    => aus S5323 wird S532

Notwendige Anpassungen für die deutsche Sprache

Da der Soundex-Algorithmus für den englischen Sprachraum entwickelt wurde, müssen erst einige Anpassungen vorgenommen werden, damit der Algorithmus auch für die deutsche Sprache gute Suchergebnisse liefert (vgl. Tabelle).

Code Buchstabe
0 a, e, i, o, u, ä, ö, ü, y, j, H
1 b, p, f, v, w
2 c, g, k, q, x, s, z, ß
3 d, t
4 l
5 m, n
6 r
7 ch

Am offensichtlichsten ist wohl, dass das deutsche Alphabet mit den Umlauten "ä", "ö" und "ü" sowie dem "ß" über Buchstaben verfügt, die im Englischen nicht existieren. Da die deutschen "Umlaute" aus phonetischer Hinsicht nichts anderes als Vokale sind, werden "ä", "ö" und "ü" auch genauso wie die anderen Vokale behandelt und bei der Codierung zunächst durch Nullen ersetzt und später eliminiert. Das "scharfe ß" wird wie das einfache "s" durch die Ziffer 2 repräsentiert.

Da es den Buchstaben "ß" nur als Kleinbuchstaben gibt, sollten die Buchstaben im ersten Schritt des Algorithmus nicht in Groß- sondern in Kleinbuchstaben umgewandelt werden.

Ein weiterer Unterschied zwischen dem Deutschen und dem Englischen liegt in der Funktion des Buchstaben "j". Während das "j" im Englischen (wie in "just" oder "join") als Zischlaut ausgesprochen wird, der im Deutschen durch die Buchstabenfolge "tsch" repräsentiert wird, erfüllt das "j" im Deutschen die gleiche Funktion wie im Englischen das "y" (vgl. "Yes" und "Ja") und muss demnach in die Gruppe der Vokale und Halbvokale fallen.

Ähnlich verhält sich der Buchstabe "w", der im Englischen als Halbvokal (wie in "what") oder als stummer Buchstabe (wie in "awesome") auftritt, im Deutschen aber als stimmhafter Gegenpart zu den Buchstaben "f" oder "v" gebraucht wird. Deshalb muss das "w" im Deutschen mit 1 codiert werden.

Eine Besonderheit der deutschen Sprache ist außerdem die Buchstabenfolge "ch", die Laute wie in "ich" oder "ach" repräsentiert. Beide Laute existieren in der englischen Sprache nicht. Da "ch" in keine der vorhandenen Kategorien passt, wird eine siebte Kategorie geschaffen.

Als weitere Anpassung an die deutsche Sprache wäre noch zu untersuchen, ob die Länge der Codewörter, die auf drei Ziffern beschränkt ist, für die deutsche Sprache angemessen ist, da im Deutschen die Wortlänge tendenziell länger ist als im Englischen.

Weitere Verbesserungen des Soundex Algorithmus

Die beschriebenen Anpassungen berücksichtigen die Unterschiede der deutschen Sprache im Vergleich zum Englischen und verbessern somit die Ergebnisse, die der Soundex-Algorithmus liefert. Dennoch birgt der Soundex-Algorithmus einige Probleme, die dadurch noch nicht behoben werden. Eines dieser Probleme ist, dass der Soundex-Code den ersten Buchstaben des Wortes als festen Bestandteil übernimmt. Das hat zur Folge, dass gleich lautende Wörter, die mit verschiedenen Buchstaben beginnen, nicht als gleich erkannt werden. Wird nach "Carina" gesucht, wird die gleich lautende "Karina" nicht gefunden.

Deswegen ist es ein weiterer Schritt, die Anfangsbuchstaben ebenfalls zu kategorisieren, so dass zumindest "C" und "K" oder "V" und "F" gleich behandelt werden. Dabei muss entschieden werden, wie grob die Kategorisierung für den Anfangsbuchstaben erfolgen soll, denn je mehr Wörter den gleichen Code erhalten, desto höher ist zwar die Wahrscheinlichkeit, dass ein falsch geschriebener Name gefunden wird, aber desto höher ist auch die Anzahl der Treffer, die mit dem ursprünglich gesuchten Namen nur noch wenig Ähnlichkeit aufweisen.

Als Kompromiss zwischen der Genauinkeit der Suche und der Wahrscheinlichkeit, einen falsch geschriebenen Namen zu finden, können mehrere Suchalgorithmen hintereinander geschalten werden, so dass zunächst nur die exakten Treffer angezeigt werden. Die Ergebnisse des Soundex-Verfahrens werden dann weiter unten in der Trefferliste angezeigt. Daran anschließend können dann noch weitere Ergebnisse von vergröberten Varianten des Soundex-Algorithmus aufgelistet werden.