nginx gegen LogJam absichern

Sicherheit und Absicherung von Servern ist wie ein eisernes Vorhängeschloss - wenn man es nicht pflegt und prüft, rostet es und wird immer weniger sicher.

Wir sind uns unserer Verantwortung sehr bewusst, dass Sie uns Ihre Daten anvertrauen. Daher sind wir immer daran interessiert unsere Server gegen gängige und neueste Attacken zu messen.

Heute ist die sogenannte LogJam Sicherheitslücke bekannt geworden. Dabei sind es vor allem die Browser, die die Sicherheitslücke aufreißen. Allerdings kann man hier in diesem speziellen Fall auch als Serverbetreiber die Anwender durch eine richtige, härtere Konfiguration des Servers schützen.

Wir nutzen nginx als unsere Edge, um unsere Backends gegenüber dem offenen Internet zu schützen. Daher haben wir uns heute, inspiriert durch LogJam, unsere nginx Konfiguration nochmal mit der Lupe angeschaut.

Die wichtigsten Punkte für die ssl Sicherheit sind in nginx die Direktiven ssl_ciphers und ssl_prefer_server_ciphers. Damit kann man unsere ssl Verschlüsselungen verbieten und den Browser und Server anweisen dem Server bei der Aushandlung des ssl Ciphers mehr zu vertrauen.

Zusätzlich ist es immer empfehlenswert ssl auch durch einen möglichst starke Diffie-Hellman Gruppe zu erzeugen. Diese wird mit dem Befehl
[crayon-5d134a6dd35d9265411181/]
erzeugt. Eingebunden werden kann diese dann über ssl_dhparam in nginx.

Überprüfen kann man die Einstellungen des Servers über weakdh.org. Schön, dass unsere Server dort "Good News!" verkünden. Wir sind gespannt, wann die nächste Sicherheitslücke unsere Server testen wird. Wir bleiben auf jeden Fall für Sie und Ihre Daten am Ball!


MTM vs. Selenium - How to run tests in parallel?

MTM:
You need to set up an environment including multiple machines in the same role. The test controller organizes the tests in buckets of 10 or 100 tests (linked to how much tests you have in your test run) and sends bucket for bucket to its agents/machines.

Selenium:
When you change the parameter in your Test Settings, then the tests will be executed in parallel. But the hub isn't able to realize that it should wait, when all instances of the grid are already in use, before broadcasting the next test request. Because of that, the next test will fail with timeout error, because the other tests took longer than the timeout.


MTM vs. Selenium - Is it possible to run automated tests on Android and iOS devices?

We already have our Test environment set up with Selenium. But we also use VSO (Visual Studio Online), which means that it could be better to change or to additionally use MS Testmanager (MTM) to organize and execute manual and automated tests. I wrote down some questions and started to analyse if any of the nice features would be lost by changeing from Selenium to MTM.

During my research I stumbled over some articles and posts about how to integrate Selenium tests into MTM. But my questions are still interesting, for the decision if I want to continue writing my tests in Selenium and then add them to MTM or to start creating them on MTM.


MTM:
You need to extend your VSO with a Plugin. Those extensions give you the opportunity to record manual tests same as on Selenium IDE - but for any mobile devices / operating systems instead of Firefox. It also has the functionality to export that recorded test into some code, which you can add on MTM.
There are various different Plugins, but none of them are for free - and I'm very happy with Selenium.

Selenium:
Here you need to set up a hub/node and connect a mobile device. Then you can record tests via Selenium IDE on Firefox and generate the test code. As last step you have to change the used WebDriver from Firefox to Android or iOS.


Command Query Responsibility Segregation (CQRS)

Command Query Responsibility Segregation (CQRS) is an architectural design pattern whose origins are attributed to Greg Young. It is based on Bertrand Meyer’s Command and Query Separation Principle (CQS), according to which a method should either be a command that performs an action, or a query that returns data, but never both at the same time.

CQRS - what it is

Following this train of thought, CQRS's key principle is to strictly separate querying from executing commands. Thus, what has been a single service in classical architectural models now is split into two separate services: a command service accommodating the command methods (methods that change the state of the system) and a query service accommodating the query methods (methods providing a view of the system's state without changing the state itself).
The result are two services which are completely independent from each other and can be designed and scaled according to their different needs.

Being able to scale each side independently is important because the number of queries is typically much higher than the number of commands. Also, when it comes to data storage, the two sides have completely different requirements. Whereas commands need to store their data in normalized tables, the data storage on the query side should be optimized to enable fast reads. Since normalized data requires complex, time-consuming queries, CQRS usually uses separate, de-normalized data tables for the query side, thus minimizing the number of joins needed for a query to get the requested information.
Of course, if we decide to work with different data storages for querying and commanding, we also need to provide a mechanism that ensures consistency across all tables.

Example: Booking system

Let me illustrate the CQRS pattern by the example of a simple booking system that allows the user to make room reservations, to get an overview about which rooms are still vacant and which not, and to view a list of all the bookings he has made.

In the system, our basic queries are getAllBookings and getBookingsByUserId. Each query is executed by its own query object and returns the information requested by the client as its search result. A query can also carry search criteria and paging options.

On the command side, we have the commands CeateBookingCommand, EditBookingCommand and DeleteBookingCommand, which are issued by the user through UI elements. Each command is modelled in a separate class and contains the minimal amount of information that is necessary for executing it (for booking a room, we would need the date, number of people, name, contact details, etc, depending on the business logic).

For each of these commands, the command model provides a separate command handler whose responsibility it is to validate the command and to make changes in the persistent data storage. If, for example, the user issues a CreateBookingCommand, the command handler assigned to deal with this kind of command will first validate the command (e.g. check if all required attributes such as date and user name are provided, check if the email address is in a correct format, etc.) and check if the requested room is still available. If everything is ok, it will then save the new booking into the database. Even though commands do not return data by definition, they may issue status and/or error messages in order to let the user know if the request was processed successfully.

CQRS_bookingSystem

Advantages

With the distinction between command and query services and the separation of querying and commanding concerns, we are able to create a solution that is optimized for either sides. The advantages of such a system can be summed up as follows:

  • Scalability: scale command and query sides independently from each other
  • Flexibility: CQRS allows the use of different representations of objects for commands and querying and is therefore more flexible than CRUD, which requires using the same classes for queries and commands
  • Better performance: separation allows optimizing each operation (e.g. quicker querying through de-normalized data storage)
  • Testability: separation of concerns leads to better testability
  • Maintainability: independent elements are easier to adapt to changing business requirements and thus easier to maintain
  • Collaborative systems: CQRS is particularly useful in collaborative systems in which multiple users operate on the shared data simultaneously. If any conflicts arise, it might not always be desirable to apply the rule 'last one wins'. CQRS allows defining more complex rules that capture the business logic.

Summary: when to use and when not to use

Even though the list of advantages above looks intriguing, not every project is suited for applying CQRS. CQRS should primarily be used in highly collaborative and complex systems in which the underlying business rules change frequently. Simple, static or non-collaborative systems, however, usually do not benefit from using the CQRS pattern.

The CQRS pattern has been very popular and widely used since it has been introduced, but that does not mean that it should be used everywhere. To put it in Udi Dahan's words, "Most people using CQRS […] shouldn't have done so". So, before employing CQRS in any project of your own, you should carefully reflect what will be gained from using it.


Checklists for improving the development process

The idea

Having recently been entrusted with the task of reviewing the team's pull requests, I realized that many things can go wrong in the process of merging branches into master if it is not handled with care. For example, it happens from time to time that changes are accidentally overwritten because someone edited an automatically generated file and forgot to forward the changes to the template from which the file is generated; or important changes get lost while resolving merge conflicts because the person resolving the conflict is not aware of the changes a colleague has made; at other times, a pull request waits to be merged into the master branch for too long (thus increasing the likelihood of merge conflicts) because someone merged another branch into it, which is not yet ready to be merged itself; or merging the PR results in a broken build because someone forgot to make sure there are no build errors in their branch. Such things just happen, and when they do, it takes a lot of time and effort to fix them.

So, being on the lookout for ways to improve the way we handle pull requests, we came up with a new idea: checklists! Checklists are already in use in many areas such as aviation, surgery or the building industry. So why not use them in our development process as well?

A brief history of checklists

Aviation checklists reach back as far as October 30, 1935, the day on which the US Army Air Corps held an aircraft competition in order to decide about the next generation of military bombers. Among the competitors was Boeing's Model 299, which had outshone the other competitors at the preceding evaluations and was considered to be the sure winner. However, as the aircraft had taken off and started to climb into the sky, it suddenly stalled, turned on one wing and exploded.

The cause of this crash was attributed to a 'pilot error'. Apparently, the pilot, being unfamiliar with an aircraft that was considerably more complex than anything he had ever flown before, had neglected a crucial step before the take-off. Because of its complexity, the news declared the aircraft to be 'too much airplane for one man to fly'. Still, the army ordered a couple of the Boeing aircrafts and had their test pilots deliberate on what to do. And they came up with a pilot's checklist with step-by-step checks for takeoff, flight, landing and taxiing.

This was the hour of birth for aviation checklists. Not even have checklist prevented countless plane crashes since then; they also have saved myriads of lives in surgeries, and helped builders schedule large-scale building projects and make sure that the resulting buildings do not crumble down because an individual missed some crucial point, just to name a few examples. With checklists, we have found a way to come to terms with tasks that are too complex for one mind alone to remember.

How to write a good checklist

Seeing that checklists can have such positive effects, the next question was how to write a good checklist for my own purpose.

The first important thing to know about checklists is that they are not intended to spell out every single step in minute detail. They are not intended to turn off your brain completely and substitute the thinking process. Rather, they assume that you are aware of what you are doing and simply want to remind you of the most critical steps. In order for a checklist to be actually used by people, it needs to be simple, short and precise. If it is vague, imprecise or too long, it will be more bothersome than helpful.

There are two different kinds of checklists: read-do lists and do-confirm lists. With the former, you carry out a task as you check it off. With the latter, you can carry out a task as you remember. In the end, you pause to run through the checklist and ensure you have not forgotten anything. Read-do lists should be employed when the order in which steps are carried out is of importance, or when a wrong step has unwanted consequences. Do-confirm lists, on the other hand, leave the people who use them more freedom to execute their task as they seem fit.

Checklists for branching and creating pull requests

With this in mind, I set out to write my own checklists. For a start, I decided to write one for creating new branches and one for creating pull requests.

The list for creating branches is a read-do list as errors here cannot easily be erased. It aims at decreasing merge conflicts and preventing the wrong branches to end up in your new branch:

checklist for branching

The list for creating pull requests is a do-confirm list that makes sure you have done what is in your power to minimize sources of errors and losses of code:

checklist for creating pull requests

Conclusion

We are going to try out these checklists in our team for the next couple of weeks. I am quite curious to see how they work and how they are accepted by the team members. Will the team members like using them? Will they consider them helpful? Will the lists fulfill their purpose and improve the process of handling pull requests? I will keep you informed about the result of this little experiment. If the checklists turn out to be successful, I am going to create more lists to use in other parts of our development process. With those steps off our minds, we can focus our brain-power on our most challenging task - producing good code.


Selenium IDE – How to verify html-table

From time to time every programmer creates a table which looks like this:

test page in browser

Please take notice, the total quantity and the sum look strange. They have to be verified.

The table is simply a set of rows. Look at source code:

source of test page

 

I’m going to sum the values from the second cell for each row and compare the calculated value with the total quantity. The same should be done with the values of the third cell to get the total amount.

In order to do that, I store the value of the second cell into a global variable ‘quantity’ in Selenium IDE
[crayon-5d134a6dd3841491347632/]
It works, but I need the value of the second cell of each rows. For this we need to loop through the lines:
[crayon-5d134a6dd3844628778190/]
Why does the increment of the index look so strange: new Number(${index} + 1) ?

Because normally “+” concatenates strings.

We get the quantity of each row now and they should sum into a different variable.

At first I define the global variable ‘totalQuantity’:
[crayon-5d134a6dd3845513024510/]
and then I sum all quantities into it in the loop:


[crayon-5d134a6dd3846482698664/]

Now I have the total quantity at the end. The same way I get the total amount into the variable ‘totalPrice’.

But there is another small problem in cells containing the price: the price has the euro sign before and the text after the price value. In order to use the value we need to get rid of all excessive characters. We can remove it as following:

price.replace(/[^0-9.,]*/g,””)

and convert a decimal separator into ‘.’ then:

price.replace(“,”,”.”)

Finally the complete script is there:

selenium ide

And, as expected, it reports an error: the total quantity must be 4 though we show 5 in the table.


Selenium Grid - How does it work

Grid

When we start one of our Selenium tests, then a request is sent to the Hub.
The Hub checks all the Nodes which are registered, for a System, that matches the desired Capabilities for the started testcase (for example, Chrome).

DesiredCapabilities

With a match, the Hub sends the Selense to the Node which is communicating with the browser and the test will be executed step by step. If there is no free slot on the matched Node, then the Hub will wait and try again until the timeout is reached.

Seleneium_Grid

Source: http://goo.gl/RwUijO

In our case the tests are structured in a way, so that it use specific URLs to reach different application systems depends where the tests will run. That means, when we start the tests local on our own developer machine, then it works with the local code and application. When we start it on the grid instead, then the Version on the Testsystem will be tested.


Iterative Performance Tuning of Web Apps

In my last blog post I described how to use Firebug's Profiler in order to find out which parts of your code are most time consuming and are responsible for the performance issues you might experience. Today I am showing you how I used the Profiler as a step in the iterative process of tuning the performance of a web app.

Steps of the iterative performance tuning process:

According to Wikipedia, the systematic tuning of performance comprises the following steps:

  1. Assess the problem and establish numeric values that categorize acceptable behavior.
  2. Measure the performance of the system before modification.
  3. Identify the part of the system that is critical for improving the performance. This is called the bottleneck.
  4. Modify that part of the system to remove the bottleneck.
  5. Measure the performance of the system after modification.
  6. If the modification makes the performance better, adopt it. If the modification makes the performance worse, put it back the way it was.
  7. start at 1. again

The Problem & Performance before the test

In my application, the performance issues concerned the searching functionality. I have a list with more than 45000 items which is searched for a search term the user can enter in a search field. The search itself uses an implementation of the Soundex algorithm (I described the Soundex Algorithm and the adoptions I made in order to use it for the German language in an earlier post) to match the search term against the items. I observed the performance to be awful especially on mobile devices.

I did not establish a numeric value to define acceptable behaviour. I just wanted it to improve significantly (being well aware that "improve significantly" is a heavy violation of the SMART principle…) so that the user does not deem the application dead while waiting for the search to finish…
Repeatedly running the Firebug profiler gave me the average of about 4500ms per searching task.

Identifying and removing the bottleneck

Using the Firebug profiler, I received the following profiling report (excerpt):

profiling report before the tuning

With the help of this report, I could easily identify the first three function calls as the bottleneck of my search. The first of them is an access to the database to retrieve all items of the list, the second the transformation of a search term into its Soundex code, and the third a filtering operation.

Generally, there are two options to deal with these functions: either to improve the function so that less time is spent for its execution, or to reduce the number of calls of that function; which of the two options is best of course depends on the function's internal logic.

I started by analyzing the first of the functions and checked where it was called. I found out that accessing the database could be prevented entirely here. Until now, the same variable was used for storing the list as it appears when it is not filtered, and for storing the list when it contains only the search results. So what happened during each search was that first, the list got emptied completely. Then, a list with all available items was demanded from the database and then filtered for the search item. I introduced a separate variable to store the list with the unfiltered items. Now the search can use this variable instead of retrieving the list from the database.

Assessing the improvement

Having made this modification, I ran the profiler again in order to check if the performance has indeed improved. As I had expected, the function that accesses the database was not invoked a single time, leading to an improvement of about 1500ms! This was a great success, so I kept the changes.

I also had a look at the other two bottlenecks. I could not improve the Soundex algorithm, nor could I reduce the amount of calls of that function. However, I was able to make some further improvement by making some minor changes to the filterBy function. Even though the modification decreased the average time spent in this function by only about 0.015ms, this accounts for quite a lot if it is multiplied with 45426, the number of calls of this function.

In total, a sorting task now takes about half the time it did before. Here is an excerpt of the profiling report after the tuning:

Profiling report after optimization


Using the Firebug Profiler for profiling web apps

I am sure every programmer is quite familiar with such situations: you are bursting with pride and self-content because your app is finally running without errors. You try it out on a mobile device for the first time, and you are thoroughly disappointed to realize that the performance of your masterpiece is just awful …
Then it is just about time for some profiling. This blog post will show you how profiling a JavaScript application can easily be done using Firebug's Profiler.

Alternative Profiling Tools

Although many other browsers provide built-in profiling tools (such as Chrome, Internet Explorer or Safari), I liked Firebug's Profiler best for its highly detailed profiling report and its way of presenting the results so that you get the most important information at one glance.

How to use the profiler

To start using Firebug's Profiler, you need to open Firebug. Select the Console tab and click on "Profile". The profiler is running now and observing all your JavaScript activity, making statistics about time consumption.

starting the firefox profiler

All you need to do now is trigger some activity. I am having performance issues with searching for a particular entry in a table in my app. So I enter a search term, start the search and wait until the search results are displayed. Then I click "Profile" again in order to stop the Profiler. Firebug now opens a huge table (well, that actually depends on the complexity of your code…) containing the profiling results. Here is an excerpt from the table I got:

profiling report

How to read the results

The great art now lies in correctly interpreting the results in order to know which part of the code is causing the trouble. Therefore it is vital to know which information is contained in the report:

In the top left corner, the profiling report specifies the total amount of time for executing the activity, as well as the total number of function calls that were involved. The table below lists every function that was called during the sort. The columns provide the following information:

  • Function: the name of the invoked function
  • Calls: how often has this function been called?
  • Percent: the percentage of time this function consumed in relation to all other functions within the sorting
  • Own time: total time spent within the function (summary of all calls)
  • Time: total time spent within the function (summary of all calls); the difference to 'own time' is that 'time' also includes the time spent within functions that were called by that function
  • Avg: average time for one call of the function
  • Min: minimal time for one call of the function
  • Max: maximal time for one call of the function
  • File: the name of the file in which the function is located and the line number of the function; a link leads directly to the file

By default the table is sorted so that those functions accounting for the highest percentage are listed first. Thus your culprits are easy to spot!

Having conducted the profiling and identified the functions responsible for the high time consumption, we need to start to analyze those functions and think of ways to improve their performance. How I used the profiling report to improve the performance of my search I will describe in a following blog post.


Selenium IDE – useful for small automation tasks

Selenium IDE is the most useful testing tool. It exists as Firefox plugin and is able to record your actions in the browser to save you from monotonously testing a web application. But sometimes Selenium IDE can help you with other automation tasks.

Here is one example: Google records every search you conduct while logged in. You can view your history here: https://history.google.com/history

If you have a lot of items in Google history and want to delete them at all, there isn’t a simple button ‘clear all’, but don’t worry. You can record these next four commands using Selenium IDE:

label | start

click | id=selector_top

clickAndWait | name=btnD

gotoLabel | start

Last but not least you finally fill the field “Base URL”: https://history.google.com/history Then start your script and enjoy!

clean google history

It looks so simple and works well, but there is one problem here: It only cleared the first page of history. And Selenium IDE hasn’t got any loop-commands. It doesn’t understand the command ‘label’ and ‘gotoLabel’.

You’ll tell yourself “Who cares?”, I’ll just replay “don’t worry”. But it gets better: Selenium IDE is an extendable tool. You can extend it with JavaScript with any functionality you want. Go to https://github.com/73rhodes/sideflow, save sideflow.js locally and then put it into the field “Selenium Core extensions (user-extensions.js)” in the Selenium IDE Options:

set sideflow script

 

Now, if you restart Selenium IDE, you’ll get a lot of loop-commands. Enjoy Selenium IDE and be happy!