Wednesday, October 17, 2018

Short-Circuit Evaluation in N4JS

Short-circuit evaluation is a popular feature of many programming languages and also part of N4JS. In this post, we show how the control-flow analysis of the N4JS-IDE deals with short-circuit evaluation, since it can have a substantial effect on the data flow and execution of a program.



Short circuit evaluation is a means to improve runtime performance when evaluating boolean expressions. This improvement is a result of skipping code execution. The example above shows an if-statement whose condition consists of two boolean expressions that combine the values of 1, 2 and 3, and its control flow graph. Note that the number literals are placeholders for more meaningful subexpressions.

First the logical and, then the logical or gets evaluated: (1 && 2) || 3. In case the expression 1 && 2 evaluates to true, the evaluation of the subclause 3 will be skipped and the evaluation of the entire condition results to true. This skipping of nested boolean expressions is called short circuit evaluation.

However, instead of skipping expression 3, expression 2 might be skipped. In case condition 1 does not hold, the control flow will continue with condition 3 right away. This control flow completely takes places within the if-condition, whereas the former short circuit targets the then block.

The reasoning behind short circuit evaluation is that the skipped code does not affect the result of the whole boolean expression. If the left hand side of the logical or expression evaluates to true, the whole or expression also does. Only if the left hand side is false, the right hand side will be evaluated. Complementary, the right hand side of a logical and expression is skipped in case the left hand side evaluates to false.


Side Effects


Risks of short circuit evaluation might arise in case a subexpression has side effects: These side effects will not occur if the subexpression is skipped. However, a program that relies on side effects of expressions inside an if-condition can be called fragile (or adventurous). In any case it is recommended to write side-effect free conditions.


Have a look at the example above. In case variable i has a value of zero, the right hand side expression i++ is executed, otherwise, it is skipped. The side effect here is the post-increment the value of i. If the value of i is other than zero, this value will be printed out. Otherwise, the value will be incremented but not printed. The control flow shows this behavior with the edge starting at i and targeting the symbol console.


Loops


Loop conditions also benefit from short circuit evaluation. This is important to know when reasoning about the all possible control flow paths through the loop: Each short circuit will introduce another path. Combining all of them makes data flow in loops difficult to understand in case of side effects in the subconditions.


Creative use of short circuit evaluation


Misusing short circuit evaluation can mimic if-statements by using expressions but without using the language feature of conditional expressions (i.e. condition() ? then() : else()). This could be used when if-statements should be executed e.g. when passing arguments to method calls, or when computing the update part of for-loops.





The picture above shows the two versions: the first uses an if-statement and the second uses an  expression statement. These two statements call the functions condition, then and end. Depending on the return value of condition, the function then is executed or not. Consequently, the printouts are either "condition then end" or "condition end", depending on the control flow.

The corresponding control flows are depicted on the right: The upper three lines refer to the if-statement, and the lower three lines to the expression statement. They reveal that the expression statement behaves similar to the if-statement. Note that the control flow edge in the last line that skips the nodes end and end() is never traversed since the logical or expression always evaluates to true.



by Marcus Mews

Thursday, September 20, 2018

Null/Undefined Analysis in N4JS

The N4JS IDE integrates validations and analyses that are quite common for IDEs of statically typed languages. However, these analyses are seldom available for dynamically typed languages like N4JS or TypeScript. In this post we present the null/undefined analysis for N4JS source code.




TypeError: Cannot read property of undefined
- Developer's staff of life


The runtime error above occurs pretty often for JavaScript programmers: A quick search on Google returned about 1.2 million for the term TypeError: Cannot read property of undefined. When constraining search results to site:stackoverflow.com the query will still yield 126 thousand results. These numbers are comparable to the somewhat similar error NullPointerException which has about 3 million hits on Google and about 525 thousand when constrained to stackoverflow.com. Some of these results are caused by rather simple mistakes that a null/undefined analysis could detect. As a result, the developer could restructure his code and remove these potential errors even before he runs his first test and hence save time.


Null/Undefined Analysis


The N4JS IDE provides static analyses to indicate problems when it detects a property access on a variable which can be null or undefined. The analysis considers all assignments that occur either through a simple assignment expression or via destructuring. Loops, conditional expressions (e.g. i = c? 1 : 0;) and declaration initializers are respected as well.

The screenshot above shows a first example where a potential TypeError is detected. Since there exists at least one control flow from v.length backwards to all reachable definitions of v,  such that one definition assigns null or undefined to v, a warning is issued telling that v may be undefined.

To make sure that the analysis will produce fast results, it is implemented within some limitations. One is that the analysis is done separately for each body of a function, method, etc. (i.e. intra-procedural analysis). Hence it lacks knowledge of variables that cross the borders of these bodies such as the return value of a nested function call. In addition, property variables (such as v.length) are not analyzed since this would require the analysis to be context sensitive to the receiver object (here v). However, these limitations are common for static analyses of statically typed languages and still allow to detect many problems regarding local variables and parameters.

Usually, the analysis makes optimistic assumptions. For instance it can happen that a local variable receives the value of a method call or another non-local variable. In this situation the analysis assumes this value is neither null nor undefined. The same is true for function parameters. Only when there are distinct indications in the source code for a value of a local variable to be null or undefined, the analysis will issue a warning.


Guards

 

Sometimes the programmer knows that a variable may be null or undefined and hence checks the variable explicitly, for instance using if (v) {...}. As a result this check disables the warning in the then-branch that complies to the execution semantics.



As shown in the screenshot above, neither at the expression w.length < 1 nor at the statement return w.length; a warning is shown. Of course, the else-branch of such a check would consequently always indicate a warning when a property of variable v is accessed. Checks for conditional expressions and binary logical expressions (e.g. v && v.length) are also supported. A reader might think: "In case w is null the expression w.length would fail." True, but in this example the analysis detects the value of w being undefined. In case null might have been assigned to w e.g. in an if-condition before, the analysis will issue a warning of w being null at the two w.length expressions.


Data Flow

There are situations where the value of a variable is null or undefined due to a previous assignment of a variable which may have been null or undefined before, like shown in the example above. Then, the null/undefined dereference problem occurs later when a property is accessed. Since the analysis respects data flow, it can follow the subsequent assignments. Hence a warning is shown at a property access indicating the null or undefined problem. Moreover, the warning also indicates the source of the null or undefined value which would be the variable w in the example above.

by Marcus Mews

Wednesday, August 29, 2018

Control flow graphs in N4JS

The N4JS IDE comes with several tools to get insights into the source code such as the AST and the control flow. In this post we present the control flow graph view.



When learning a programming language, we probably start with a small Hello World! program. From there on, we learn not only language keywords and libraries, but also get an implicit understanding in which order statements and expressions are executed. This order is called control flow. For instance, we learn the effects of if and for statements which are also called control statements, since they have a big influence on the execution order. The most difficult statement in this respect is probably the try-finally control statement, which is explained later.


Hello World


function f() {
  console.log("Hello World!");
}

Let's have a look at the Hello World! from a control flow perspective, first. The source code of Hello World! is simple, but it already consists of three important elements:
  • the method call console.log,
  • the nested argument "Hello World!", and
  • the function body of f() which contains the two elements mentioned above.
The function body is also called control flow container.


The image above shows the control flow graph of the Hello World! example. The method call is separated into the receiver and its property log. The next element the control flow goes to is the argument "Hello World!" until it reaches the method call of log as the final element.


Loops


The example above showed the succeeding control flow of some code elements. This control flow gets more interesting in statements that introduce branches such as loop statements do. In the example below, the control flow of a for-loop is shown. To indicate the start and end of the function, and also the body of the loop, the function calls start(), loop() and end() are used.

function f() {
  start();
  for (var i=0; i<2; i++) {
  loop();
  }
  end();
}


The control flow of the loop example shows branches and merges. After the condition i<2, either the body is entered via the edge named LoopEnter, or the loop is exited. In case the body was entered, the control flow first targets the call to loop() and then goes back to the entry of the condition.


Try-Finally


Finally blocks have tricky semantics since they can be entered and exited in two specific ways: normally and abruptly. Abrupt control flow occurs after a return, continue, break or throw statement. These will introduce a jump that targets either the end of the function or the entry of a finally block. In case the control flow jumps to a finally block, this finally block will be executed normally. However, since the block was entered abruptly, it will exit abruptly again. This means, that there will be a second jump from the exit of the finally block to either the end of function or the entry of the next finally block.


The following example shows this behaviour by using some dead code elements, which have a grey background colour in the graph image. These dead code elements are not reachable, since there is no normal control flow path that exits the try block.


function t() {
  "start";
  try {
  2;
  return;
  3;
    } finally {
  "finally";
  }
    "end";
}


Final catch


In case you ever wondered how a thrown exception can be caught without using a catch block, have a look at the final example. It is true that once a finally block was entered abruptly, it can only be exited abruptly. However, the kind of abrupt control flow might be changed. In the example below, it is changed from throwing an exception to breaking the loop. Of course, after the loop was exited due to the break statement, the control flow is normal again and the thrown exception remains without effect. Hence, the last statement "end" is executed which would have been skipped otherwise.



function t() {
  "start";
  do {
    try {
      2;
      throw "exception";
    } finally {
      break;
    }
  } while (4);
  "end";
}














by Marcus Mews

Friday, May 19, 2017

N4JS Becomes an Eclipse Project

We’re proud to announce that N4JS has been accepted as an Eclipse Project and the final official steps are underway. Our team have been working very hard to wrap up the Initial Contribution and are excited to be part of Eclipse. The project will be hosted at https://eclipse.org/n4js, although this currently redirects to the project description while our pages are being created. In the meantime, N4JS is already open source - our GitHub project pages are located at http://numberfour.github.io/n4js/ which contains articles, documentation, the source for N4JS and more.

Some background information about us:
N4JS was developed by Enfore AG, founded in 2009 as NumberFour AG by Marco Boerries. Enfore’s goal is to build an open business platform for 200+ million small businesses and to provide those businesses with the tools and solutions they need to stay competitive in a connected world.

Initially, JavaScript was intended as the main language for third-party developers to contribute to our platform; it runs directly in the browser and it’s the language of the web! One major drawback is the absence of a static type system; this turned out to be an essential requirement for us. We wanted to ensure reliable development of our platform and our own applications, as well as making life easier for third-party contributors to the Enfore platform. That’s the reason why we developed N4JS, a general-purpose programming language based on ECMAScript 5 (commonly known as JavaScript). The language combines the dynamic aspects of JavaScript with the strengths of Java-like types to facilitate the development of flexible and reliable applications.

N4JS is constantly growing to support many new modern language features as they become available. Some of the features already supported are concepts introduced in ES6 including arrow functions, async/await, modules and much more. Our core team are always making steady improvements and our front end team make use of the language and IDE daily for their public-facing projects. For more information on how the N4JS language differs from other JavaScript variants introducing static typing, see our detailed FAQ.

Why Eclipse?
For us, software development is much more than simply writing code, which is why we believe in IDEs and Eclipse in particular. We were looking for developer tools which leverage features like live code validation, content assist (aka code completion), quick fixes, and a robust testing framework. Contributors to our platform can benefit from these resources for their own safe and intuitive application development.

We tried very hard to design N4JS so that Java developers feel at home when writing JavaScript without sacrificing JavaScript’s support for dynamic and functional features. Our vision is to provide an IDE for statically-typed JavaScript that feels just like JDT. This is why we strongly believe that N4JS could be quite interesting in particular for Eclipse (Java) developers. Aside from developers who are making use of N4JS, there are areas in the development of N4JS itself which would be of particular interest to committers versed in type theory, semantics, EMF, Xtext and those who generally enjoy solving the multitude of challenges involved in creating new programming languages.

What’s next?
While we are moving the project to Eclipse, there are plenty of important checks that must be done by the Eclipse Intellectual Property Team. The Initial Contribution is under review with approximately thirty Contribution Questionnaires created. This is a great milestone for us and reflects the huge effort involved in the project to date. We look forward to joining Eclipse, taking part in the ecosystem in an official capacity and seeing what the community can do with N4JS. While we complete these final requirements, we want to extend many thanks to all at Eclipse who are helping out with the process so far!

Thursday, November 3, 2016

BoF @ ECE 2016: Eclipse and AsciiDoc

At NumberFour, we use AsciiDoc for our specifications and documentation. We are even working on producing AsciiDoc API documentation from JSDoc-like comments (similar to JavaDoc) for our JavaScript language N4JS. Since we ran into a couple of problems with AsciiDoctor, I thought that it might be interesting to learn what other people in the Eclipse Community think about AsciiDoc(tor) and if we might be able to "join forces" to overcome certain obstacles. So I organized a "Birds of Feather" about "Documentation with AsciiDoc and Eclipse" at EclipseCon 2016 Europe. We were about 10 participants and this blog post serves more or less as a kind of meeting minutes of this BoF combined with our findings at NumberFour.

AsciiDoc is a lightweight markup language, similar to Markdown. The original AsciiDoc was written in Python, but Asciidoctor, probably the most popular tool nowadays, is written in Ruby with a variant AsciidoctorJ which provides Java support (via JRuby) and Asciidoctor.js for JavaScript (via Opal). From an Asciidoc document (.adoc) you can either create HTML, PDF or even EPub. Unfortunately, the PDF support is lacking some major features (e.g., footnotes are not supported yet), but there is a common workaround to produce DocBook XML files from AsciiDoctor and then use FOP to eventually create a beautiful PDF. Lars Vogel wrote a nice tutorial about AsciiDoc and AsciiDoctor. One of the great things about AsciiDoc is that GitHub provides basic support for AsciiDoc as well!

Writing AsciiDoc

The first question is how to write AsciiDoc. One solution is to use stand-alone editors such as Sublime (there is an AsciiDoc plug-in for syntax highlighting and coce completion which, together with the preview plug-in, is a really great solution).

But usually, you do not want to switch editors when you are already using Eclipse. Fortunately, there's an existing AsciiDoc editor which is part of Mylyn Wiki Text. This editor has some syntax highlighting, an outline view and even a preview.

This editor seems to be a good starting point for further support of AsciiDoc on the Eclipse platform, although it might have some shortcomings with regard to features and design. Regardless, the BoF participants (including Torkild, a Mylyn Docs committer) decided to use the mylyn-docs developers mailing list for further communication about AsciiDoc and Eclipse.

Apparently, having a good editor, possibly even with WYSIWYG or at least WYSIWYM support is one of the most important things required.

Building AsciiDoc

There are several ways of building AsciiDoc. The most simple one is about using the AsciiDoctor tools directly from the command line. There is also a Maven plugin provided by the AsciiDoctor project.

We haven't discussed that topic any further though. It seems as if the existing tools are sufficient even though there are some shortcomings when using custom macros as described below. You may have a look at the N4JS documentation pom.xml to get an idea of how this could look like.

Converting to AsciiDoc

During the BoF, the question came up how to convert existing documentation written in other formats to Asciidoc. One solution for that is Pandoc, which we (at NumberFour) are using to convert our LaTeX specification to Asciidoc. Jeremie Bresson has already blogged about the BoF and even provided a solution for using Mylyn Wikitext to convert Eclipse MediaWiki to Asciidoc. Thank you very much, Jeremie!

Using AsciiDoc as a Single Source

I already mentioned above that you can generate HTML and PDF from AsciiDoc. However, HTML is not HTML. We use AsciiDoc to generate

  • the public web site (i.e., gh-pages)
  • the Eclipse help
  • the PDF specification

from a single asciidoc file. To give you an idea, have a look at an adoc source file and the generated web page. If you download the N4JS IDE, you can also have a look at the generated Eclipse help containing the same content. The PDF is not publicly available yet, we are currently working on migrating our language specification from LaTeX to AsciiDoc.

In order to generate Eclipse help, a table of contents file is required. At the moment, we generate them via the geneclipsetoc-maven-plugin Maven plugin, see our pom.xml for our configuration.

You may also use the Mylyn Docs to generate code, which is described in the Mylyn Docs Eclipse help. This also provides support for generating EPub, the first steps towards this support has already been described 2011 in Torkild's blog! It’s now a stable part of Mylyn Docs and has been for a few years. Currently it is in maintenance mode as there are few bugs and there is not much to add – apart for EPUB 3.0 support. You can find an EPUB examples (and presentations about Mylyn Docs in general) on Torkild's GitHub page.

Customizing AsciiDoc

Although AsciiDoc provides a lot of features ("macros" in Asciidoc terminology) for technical documentation, such as source blocks, warning blocks etc., we (at NumberFour) missed some functionality. We required additional support for

  1. definitions -- a typical feature required in specifications
  2. BibTex cites and bibliographies -- we already have a large BibTex database which we want to reuse
  3. todos -- well, also a typical feature required in specifications and documentations
  4. links to source code -- e.g., for adding links to either GitHub, or, in case of Eclipse help, to the files in the workspace
  5. larger documents, e.g., includes similar to LaTeX's chapter folders

One problem we found is that Ruby-based custom macros calling native code cannot be used with AscidoctorJ. E.g., there already exists a custom Ruby macro which provides support for BibTex, but this does not work with AscidoctorJ. We want to write the macros in Java, since we plan to provide some editor support in Eclipse as well -- and then Java would be the better solution. Thus we already wrote our own Java-based, BibTex macro (fixing some smaller AsciidoctorJ problems on the way) and we are also working on solutions for the other topics. If you are interested in that, or if you have written Java-based macros for AsciiDoc as well -- let me know! We haven't open sourced our custom macros yet since they are still under (heavy) development. But we will probably do that once they are in a state to make them public.

Edit: Updated description of EPUB support -- thank you Torkild for the information!

Note: This blog post was initially posted at jevopi's developer blog (which is listed on Planet Eclipse). Since it is highly related to the N4JS project, we re-posted it here again.

Wednesday, May 11, 2016

Test-Driven Development with N4JS

The N4JS IDE validates and checks code as you type, making it a powerful tool for writing type safe applications that that are predictable, reliable and easily maintainable over time. This means that problem-solving takes place before code is deployed and your applications are safely exported.

As test-driven development is a key technique for the development of large-scale projects, the N4JS IDE is bundled with a robust test framework called Mangelhaft. Mangelhaft is an xUnit-like framework with an engine for executing tests and will report results to the IDE while running test suites.

Testing with Mangelhaft in the N4JS IDE 

While Node.js projects written with N4JS can easily be checked and tested with Mangelhaft within the IDE, the release of Mangelhaft on npm means that your Node.js projects will stay stable when deployed on a Continuous Integration server.

Using Mangelhaft with Travis CI

Using the example project bundled with the N4JS IDE, we can demonstrate how to use Mangelhaft with a Travis build. In the Travis .yml file, we define the language of the build as node_js. Currently, Java8 must also be installed as a requirement for the build:

    sudo: required
    language: node_js
    before_install:
      - sudo apt-get -qq update
      - sudo apt-get install -y oracle-java8-set-default
    node_js:
      - "5"
    services:
      - mongodb
  

Mangelhaft can either be installed globally ( npm install -g n4js-node-mangelhaft ) or integrated with a build by including Mangelhaft in the devDependencies of a project and exporting as an npm package. Mangelhaft will be downloaded and installed automatically when the Travis build begins:

Installing Mangelhaft on Travis CI Server

Default test scripts described in the package.json can then be run with the npm test command which calls n4js-mangelhaft.

Test results in Travis CI

A full list of available options can be listed with the command n4js-mangelhaft -h

Using Mangelhaft with Jenkins

Mangelhaft can also be integrated with a Jenkins job and fully supports detailed xUnit test results. View the status of your tests over time with the Jenkins UI and benefit from individual test reports:

n4js-mangelhaft \
      …
      --xunitReportFile ./build/tasks-test-report.xml \
      --xunitReportName tasks-test-report \
      --xunitReportPackage n4js.example.tasks.tests \
      …
  
Mangelhaft Test Results in Jenkins UI 
Mangelhaft Test Results in Jenkins UI

Generate comprehensive xUnit test data as frequently as required and drill down to the root of any issues that arise:

Individual Test Results in a Jenkins Build

Type Safe Workflow with Mangelhaft

Running npm tests with Mangelhaft ensures Node.js projects created with N4JS stay stable with errors detected and located quickly and easily. Download the N4JS IDE and begin creating type safe Node.js projects with the peace of mind of CI tests made possible with Mangelhaft.

Resources referred to in this post

Sunday, March 13, 2016

N4JS Goes Open Source

Today NumberFour AG announced "N4JS,  An Open-Source Language and IDE for high

N4JS is built for JavaScript developers who seek to build large-scale and maintainable server-side JavaScript as well as Java developers who want to use JavaScript with familiar tool support, early validation and other Java benefitsIt bridges the strengths of JavaScript and Java; the result is a typed JavaScript superset that is dynamic, flexible and type-safe.

N4JS Feature Highlights

The N4JS language and its IDE allow for high-quality JavaScript development of large Node.js projects. Here is a list of some of its features:
  • super set of ECMAScript 5; most of ECMAScript 2015
  • transpiles to V8-compatible ECMAScript (Node.js version 5)
  • Java-like nominal types
  • TypeScript-like structural types
  • generics similar to Java 8
  • async/await (TC39/ECMAScript proposal) with support for legacy code
  • dependency injection using annotations similar to JSR330/Google Guice
  • built-in support for testing, including xUnit-like test framework "mangelhaft"
  • Node.js support: automatically download npms, export to npm

Visit the N4JS webpage to learn more about N4JS and download the latest N4JS IDE.

N4JS for Eclipse-Developers

You are a Eclipse plugin developer looking for a high-quality Java-based ECMAScript parser.
You develop DSLs wich are to be combined with JavaScript environments?

N4JS provides an Xtext-based ECMAScript 2015 parser. The created AST and type model are based on EMF, allowing for integration with other Eclipse modeling tools. Set-up is made easy due to provided Oomph set up script.The N4JS transpiler and the IDE are Eclipse-based.

Visit the N4JS GitHub page to access the source code of the N4JS IDE and the transpiler.

The N4JS Developer Blog

In this blog, the N4JS developer team at NumberFour will write about N4JS. Topics will include

  • new features
  • explanation of existing features
  • discussion of language concepts
  • description of design and implementation of the N4JS IDE and the transpiler

We are looking forward to your feedback! You can either leave comments, post to our mailing list, create issues, or, last but not least, create pull requests!