Tuesday, February 9, 2016

Unboxing Packages: source_span

One of the best aspects of a package ecosystem is its ability to foster language-wide conventions without direct involvement from the SDK. Packages work together and build on one another to create something more than any of them could be on their own. The glue that holds this structure together is made of little packages that establish these conventions, that provide a shared language for the ecosystem as a whole to use.

One of these glue packages that I find most useful is source_span. This package began its life as part of the more specialized source_maps package, it quickly became clear that the idea of referring to chunks of code was much more broadly applicable. I factored it out, tweaked the API to make it more general, and over time added a few additional features. The result is the package I’ll talk about today.

The basic idea underlying source_span is that there should be a straightforward and consistent way for packages to refer to spans of text. This is useful for associating spans within a source map, but it’s also useful for emitting nice errors when parsing text. The den package uses them to edit YAML without disrupting the existing formatting.

The structure of the SourceSpan class is very straightforward. Here are the most important parts:

class SourceSpan {
  Uri get sourceUrl;
  SourceLocation get start;
  SourceLocation get end;
  String get text;
}

class SourceLocation {
  Uri get sourceUrl;
  int get offset;
  int get line;
  int get column;
}

A span knows where it starts, where it ends, and what text it covers; a location knows its offset in the text, as well as its line and column numbers (all of which are zero-based). Both of them know the URL of the file they refer to. That’s the heart of the package. Now you know. Article over. Good job, Natalie!

Just Kidding

Okay, there’s more to it than that. For starters, there are some handy methods on those classes. SourceSpan.length, for example, tells you how long a span is so you don’t have to do the math (or call .text.length) manually. SourceLocation.distance() tells you the number of characters between two locations. And both classes are Comparable—locations are ordered by their offsets, and spans are ordered by their start locations.

But my favorite method is SourceSpan.message(). This formats a message so that it’s clearly associated with the given span. It sounds simple, but it’s a big part of what makes the package so nice to use, since it provides such a nice user experience for so little effort. Here’s an example of what it looks like:

Error on line 24, column 15 of pubspec.yaml:
Invalid version constraint: Expected version number after "^" in "^1.2.".
  pub_semver: "^1.2."
              ^^^^^^^

Dart users might recognize this particular error, because it comes from pub, which was one of the first users of the source_span package. Here’s how that error was produced:

stderr.writeln("Error on ${span.message(error.message, color: true)}");

That’s all you need for beautiful messaging. They can be about errors or anything else associated with a particular span of a file.

Creating Spans

If you want, you can construct spans and locations by calling new SourceSpan() and new SourceLocation(), but when you’re parsing a bunch of text in a single file, that’s a lot of work (and memory-inefficient to boot). You’re much better off using a SourceFile to create them. You can create one by passing in the file text and URL to new SourceFile().

A SourceFile represents a single source file, and spits out spans and locations for that file. SourceFile.span() gets you a span for a chunk of text, and SourceFile.location() gets you a location at a specific offset. You can also get assorted metadata about parts of the file without creating extra objects by calling SourceFile.getLine(), SourceFile.getColumn(), SourceFile.getOffset(), and SourceFile.getText().

Implementation and Efficiency

A SourceFile does more than just fill in the blanks for the normal span and location constructors. Because it has access to the entire file, it’s able to do some clever tricks to make the spans more efficient, especially in terms of memory. And when you’re parsing a large file and creating a span for every node, memory usage can matter a lot.

First of all, a SourceFile’s spans don’t actually store SourceLocation objects. Instead, they store the start and end offsets as integers and create SourceLocations on the fly when the corresponding getters are called.

The locations’ line and column information is also determined on the fly. SourceFile keeps an internal list of the offsets for each newline character in the file, which is quite compact. But it also means that looking up the line or column for an offset requires a binary search, which is O(log(n))—not slow, but not super fast either. There’s some caching which mitigates this when looking up multiple nearby offsets, but it’s still important to keep the worst case in mind.

As a user, this all means that you should be careful when looking up spans’ locations, and especially when accessing those locations’ line and column numbers, in code that needs to be high-performance. You don’t need to avoid these calls entirely, but make sure you aren’t calling them dozens of times in a core loop.

SourceSpanWithContext

I admit it: I cheated a little bit when I was talking about SourceSpan.message() earlier. The error message I gave as an example contained extra text from the pubspec—the text “pub_semver:” wasn’t covered by the span, and thus a plain SourceSpan wouldn’t know to print it. It would have printed a less-useful message that just included "^1.2.".

However, often spans do have that extra context, and when they do it makes for a much better message. That’s what SourceSpanWithContext is for. It’s a SourceSpan with one extra field: SourceSpanWithContext.context, which returns the full line containing the span’s text. This is enough to produce the extra-helpful message I included above.

FileSpan

Because a SourceFile always has access to extra context information, all the spans it produces are SourceSpanWithContexts. But more than that, they’re FileSpans. This is a special kind of span that’s only generated by a SourceFile, and in fact has a getter for the original file. It also has a few superpowers of its own.

All spans have a SourceSpan.union(), which combines two spans into one. However, it requires the spans to be adjacent or overlapping; otherwise, there would be no way to determine the text of the resulting span.

But that’s no problem for FileSpans, because they know the full contents of the file. So they support FileSpan.expand() as well, which only takes another span from the same file and returns a new span that covers the entire distance between them. This is really useful when constructing spans for use in a source map.

Span Exceptions

One of the most important uses of spans is to indicate to the end user where an error was discovered, so it makes sense that there would be a standard way to attach them to exceptions. The source_span package provides two different exception types: SourceSpanException represents a general exception that has a span attached, and SourceSpanFormatException also implements FormatException since so much code that uses spans deals with parsing as well.

These exceptions’ APIs are pretty simple. They provide access to the span, of course, and their toString() methods use SourceSpan.message() to format the error message. You can even pass in a color parameter to print it with nice terminal colors on Linux and OS X.

Despite their simplicity, though, the exception classes are powerful because they allow unrelated packages to work together. Any package can throw SourceSpanExceptions as a matter of course, and any package can catch them and format them nicely.

The Spannotation Pattern

“Spannotation” is less of a well-known design pattern than something I named just now as I wrote this, but that doesn’t make it any less useful when dealing with parsed data. The idea is very straightforward: as you parse text into higher-level data structures, keep a span along to remind you of where the data came from.

As an example, let’s look at pub. It initially parses a pubspec into a class that stores the original fields as a YamlMap, which has spans attached to every node it contains:

class Pubspec {
  final YamlMap fields;

  Pubspec.parse(String contents, {sourceUrl})
      : fields = loadYamlNode(contents, sourceUrl: sourceUrl);
}

When a caller wants to know how the transformers are set up, they’re parsed from these fields into TransformerConfig objects:

class Pubspec {
  List<Set<TransformerConfig>> get transformers {
    return fields['transformers'].nodes.map((phase) {
      return phase.nodes.map((transformer) {
        var name = transformer.nodes.keys.single;
        var config = transformer.nodes.values.single;

        // Keep track of the transformer name's span.
        return new TransformerConfig(name.value, config, name.span);
      }).toSet();
    });
  }
}

TransformerConfig keeps track of the span that was parsed from the transformer’s name:

class TransformerConfig {
  final String name;
  final Map config;
  final SourceSpan span;

  TransformerConfig(this.name, this.config, this.span);
}

That way, when an error is detected relating to this transformer, we can attach the span to that error to make the error much nicer.

Future<Transformer> loadTransformer(TransformerConfig config) async {
  try {
    return new RemoteTransformer(spawnTransformerIsolate(config));
  } catch (error) {
    // Attach the configuration's span to the error so the user knows what
    // transformer caused it and where to configure it.
    throw new SourceSpanException(error.toString(), config.span);
  }
}

Spanning the Gap Between Packages

I love package ecosystems, and the source_span package embodies many of the aspects I love best. It’s pretty small and pretty simple, but it serves an important role. It provides a consistent way for packages to expose and interact with sections of source text, and it does so in a way that makes the end user’s life better as well.

Come back again in two weeks when I go over a package that, among other things, makes it really easy to create spans.

Friday, January 29, 2016

Dart 1.14 continues to improve core APIs and tools

Dart 1.14 is now available. This release contains a number of additions and improvements across our core libraries and our tools.

Symbolic links for package resources can cause problems, especially on Windows. Since Dart 1.12, we’ve been working to support a Package Configuration File. The goal of this work is to provide a better model for package access. Dart 1.14 contains new APIs in both dart:io and dart:isolate to provide code access to the configuration file if it’s being used. You should consider these APIs experimental as we update the test package and pub to validate the implementation.

Access package assets easily and reliably

Developers frequently need to directly access assets within packages. In the past this has lead to confusion when supporting the many ways in which a Dart program can be executed. Along with the new Package Configuration APIs added in Dart 1.14, we’ve also released a resource package. This package makes accessing assets within a package straightforward.

import 'package:resource/resource.dart' show Resource;

main() async {  
  var resource = new Resource("package:foo/foo\_data.txt");  
  var string = await resource.readAsString();  
  print(string);  
}

And more…

The SDK changelog has details about all of the updates in Dart 1.14. Get it now.

Tuesday, January 26, 2016

Unboxing Packages: collection

I often find myself working with a class of Dart packages I like to think of as core library expansion packs. They aren’t the splashiest, they don’t produce cool demos, but they’re extremely useful. These expansion packages, which are always named after a dart: library, build upon that library’s design and functionality to provide utility classes and fill in missing features.

Actual core libraries—the things you get when you write import "dart:..."—are very useful because they’re always there, but they’re also really painful to change. Exactly because they’re always there, they have much stronger stability guarantees than the rest of the ecosystem. It’s dangerous to even add a new class because it might conflict with a user’s class name and break existing code. And forget about adding an instance method. What if someone was implementing that class as an interface?

For those reasons, the core libraries are kept as slim as possible. So what happens when we want to add a new collection type, or a class for manipulating streams? That’s exactly what expansion packs are for. Because they’re versioned packages, they can safely be added to and modified without breaking the world. But we can still keep them to the high standard of API design and stability to which we hold the core libraries.

The best way to understand the general idea is to look at a particular example. Let’s dive into the collection package, the oldest core expansion and one of the oldest Dart packages period. This package has a bunch of classes and mixins to help implement the various (rather hefty) collection interfaces, as well as advanced equality support, and a few specialized functions and collection types.

Wrappers

A wonderful feature of Dart’s type system is how easy it is to expose something that users see as a normal List or Map, but that has customized behavior or representation internally. The wrappers provided by the collection package are a key component of making this work. They make it as easy as extends to write a class that modifies the behavior of a collection.

The simplest examples are the delegating classes: DelegatingIterable, DelegatingList, DelegatingMap, DelegatingSet, and DelegatingQueue. These classes wrap an object of the corresponding type and, by default, forward all methods to that object. But you can extend them to override specific methods:

/// A delegating list that improves the `toString()` to only include modified
/// flags.
class FlagList extends DelegatingList<VMFlag> {
  FlagList(List<VMFlag> flags) : super(flags);

  String toString() {
    return "[" + this.where((flag) => flag.modified).join(", ") + ", ...]";
  }
}

The unmodifiable wrappers also come up a lot in practice. UnmodifiableListView and UnmodifiableMapView were so popular that they were actually brought into dart:collection itself, but UnmodifiableSetView is only in the package. These classes are like the delegating wrappers, but make all methods that would modify the collection instead throw UnsupportedErrors. They’re great for writing classes that expose collections that shouldn’t be modified externally:

class Engine {
  List<LiveTest> get liveTests => new UnmodifiableListView<LiveTest>(_liveTests);
  final _liveTests = <LiveTest>[];

  // The class internally modifies _liveTests, but external users can't.
}

There are also unmodifiable mixins that complement the views: UnmodifiableListMixin, UnmodifiableMapMixin, and UnmodifiableSetMixin. You can mix these in to your own, non-view collections to easily make an unmodifiable version.

Equality

Most of the time, Dart’s == is perfectly good for telling if two objects are the same. But sometimes, you need more, and the collection package has your back. It defines an Equality<T> interface that defines two main operations: whether two objects of type T are equal, and what the hash code for an object of type T is.

The most useful type of equality after == is DeepCollectionEquality. This compares supported collections (iterables and maps) based on their contents, whereas == only returns true if they’re the exact same object. There’s even a new DeepCollectionEquality.unordered() constructor if you don’t care about ordering.

void assertEqual(actual, expected) {
  if (const DeepCollectionEquality().equals(actual, expected)) return;
  throw "Expected $actual to equal $expected.";
}

There’s also DefaultEquality, which is the same as ==, and IdentityEquality, which always checks if two objects are the same. These aren’t very useful on their own, but they’re important for constructing custom equalities. Which, by the way, is a thing you can do! The equality classes are constructed to allow you to build new equalities out of old ones.

For example, new DeepCollectionEquality() optionally takes an Equality parameter that it uses to compare non-collection objects. MapEquality takes two different Equality paramters, one for keys and one for values. And MultiEquality takes a whole list that it tries in order.

Specialized Collections

My favorite part of the collection package is its original collection classes. Partly this is because I wrote a bunch of them and I have a mother’s pride, but it’s also because a good collection does what you need and does it well, and that’s just cool.

CanonicalizedMap

Most of these classes are implementations of existing interfaces, but with useful specialized behavior. The CanonicalizedMap class, for example, is a map that converts keys to a canonical form. You can use it to do things like make a case-insensitive map for, say, HTTP headers.

class CaseInsensitiveMap<V> extends CanonicalizedMap<String, String, V> {
  CaseInsensitiveMap()
      : super(
          // This callback determines the canonical form of the user's key.
          (key) => key.toLowerCase(),

          // Null keys aren't allowed. You can't call null.toLowerCase()!
          isValidKey: (key) => key != null);
}

void main() {
  var headers = new CaseInsensitiveMap<String>();

  // This canonicalizes the key to "content-type", then stores the value.
  headers["CONTENT-TYPE"] = "application/dart";

  // This reads back the same value, since the canonical key is the same.
  print(headers["Content-Type"]);
}

When carefully thinking about API boundaries, which everyone should do all the time forever, it occasionally becomes clear that the collection a class uses internally doesn’t provide the best interface for users of that class. You can always copy the data into a new form when the user tries to read it, but in some circumstances that’s not as efficient as it could be.

MapKeySet and MapValueSet

One of those circumstances is when you’ve got a map internally, but you want to expose it as a set. Sets and maps both ensure that all elements or keys are unique, after all. That’s what the MapKeySet and MapValueSet classes are for. They each wrap a map and expose its keys and values (respectively) as a set, without actually copying those contents. This means that if the underlying map is updated, that update is automatically reflected in the wrapper as well.

MapKeySet is the simpler of the two: it’s literally just a set of all the keys in the map. It’s unmodifiable, because there’s no way to add a key to the map without a corresponding value, but it’s great for exposing an internal map’s keys.

MapValueSet exposes the values of a map as a set instead of the keys. You can’t efficiently figure out whether a value is in a map without some help, so this takes an extra callback that takes a (potential) value and returns the key that corresponds to it. And of course, that only works if it’s possible to compute the key for a value at all. If you have such a function, though, you get an efficient value set. And it can even be modified!

class DoofyDatabase<T> {
  // Internally represent the database as a map so it can be indexed by ID.
  final _map = <int, Doof>{};

  // Externally expose the database values as a set.
  final Set<Doof> allDoofs;

  DoofyDatabase() : allDoofs = new MapValueSet(_map, (doof) => doof.id);

  Doof doofById(int id) => _map[id];
}

PriorityQueue

A priority queue is one of those classic data structures you learn about in school, and PriorityQueue is the Dart implementation. Well, actually, HeapPriorityQueue is the implementation. PriorityQueue should probably have a factory constructor—let me file an issue about that real quick. If one of you lovely readers wanted to send in a pull request to fix that, I’d personally review and merge it.

A priority queue is a collection that’s efficiently kept constantly ordered even as objects are added and removed. The exact ordering depends on the Comparator you pass to the HeapPriorityQueue constructor, but by default it just assumes the contents are Comparable and calls a.compareTo(b). PriorityQueue implement any of the more common collection interfaces, but you’ll recognize most of the methods from Set and Queue.

QueueList

QueueList has all the powers of both Queue and List! The normal queue interface doesn’t have an [] operation because some implementations can’t support it efficiently. But other implementations actually use a list as the underlying data structure, which makes [] very efficient. QueueList takes advantage of this fact to implement both interfaces at once.

The result is a data structure that’s efficient at everything it does. Unlike a list, elements can be added and removed from the beginning in O(1) time. Unlike a queue, elements can be accessed from the middle in O(1) time. It’s a beautiful thing.

Collecting the Goods

You can find the source code for collection on GitHub, where you should feel free to send pull requests and add your own awesome collection classes. And next time your API needs to expose a set that can’t be modified or needs to remove an element from the beginning of a collection and access elements in the middle, you can download it with pub.

Join me in two weeks when I talk about how to make your parse errors look amazing.

Tuesday, January 12, 2016

Unboxing Packages: stack_trace

Editor's note: This is the first post in a new series by Natalie going in-depth on Dart packages and how they're used.

If you’ve written any Dart code at all, you’ve definitely used tools that use the stack_trace package. It was originally written for use in pub, and the test package uses it for all the stack traces that tests generate. But you don’t just have to rely on other tools to use it for you—you can use it yourself!

The package has three main use-cases, all of which are important and useful. Originally, it was just designed to make existing stack traces prettier and easier to read. A big part of that was parsing stack traces into a data structure, so its secondary purpose became programmatically manipulating stack traces. Then once zones were created, it added support for tracking stacks across asynchronous calls.

Making Traces Pretty

Let’s face it: the Dart VM’s built-in stack traces aren’t nice to look at, and they don’t present information in a way that’s easy for humans to read. Let’s take a look:

#0      Object.noSuchMethod (dart:core-patch:1884:25)
#1      Trace.terse.<anonymous closure> (file:///usr/local/google-old/home/goog/dart/dart/pkg/stack_trace/lib/src/trace.dart:47:21)
#2      IterableMixinWorkaround.reduce (dart:collection:29:29)
#3      List.reduce (dart:core-patch:1247:42)
#4      Trace.terse (file:///usr/local/google-old/home/goog/dart/dart/pkg/stack_trace/lib/src/trace.dart:40:35)
#5      format (file:///usr/local/google-old/home/goog/dart/dart/pkg/stack_trace/lib/stack_trace.dart:24:28)
#6      main.<anonymous closure> (file:///usr/local/google-old/home/goog/dart/dart/test.dart:21:29)
#7      _CatchErrorFuture._sendError (dart:async:525:24)
#8      _FutureImpl._setErrorWithoutAsyncTrace (dart:async:393:26)
#9      _FutureImpl._setError (dart:async:378:31)
#10     _ThenFuture._sendValue (dart:async:490:16)
#11     _FutureImpl._handleValue.<anonymous closure> (dart:async:349:28)
#12     Timer.run.<anonymous closure> (dart:async:2402:21)
#13     Timer.Timer.<anonymous closure> (dart:async-patch:15:15)

Look at those numbers that don’t add any information. Look at all those lines of internal SDK calls that the user doesn’t care about. Look at those gargantuan absolute file: URLs where relative paths would do. Look how little alignment there is across lines. This is messy. And it’s even worse if you’re compiling to JavaScript; then you have to deal with a bunch of different messy formats!

The initial motivation for the stack_trace package was just to print stack traces in a way Bob and I liked to read while we were writing Pub. And it made a huge difference just by aligning columns and making paths relative:

dart:core-patch 1884:25                     Object.noSuchMethod
pkg/stack_trace/lib/src/trace.dart 47:21    Trace.terse.<fn>
dart:collection 29:29                       IterableMixinWorkaround.reduce
dart:core-patch 1247:42                     List.reduce
pkg/stack_trace/lib/src/trace.dart 40:35    Trace.terse
pkg/stack_trace/lib/stack_trace.dart 24:28  format
test.dart 21:29                             main.<fn>
dart:async 525:24                           _CatchErrorFuture._sendError
dart:async 393:26                           _FutureImpl._setErrorWithoutAsyncTrace
dart:async 378:31                           _FutureImpl._setError
dart:async 490:16                           _ThenFuture._sendValue
dart:async 349:28                           _FutureImpl._handleValue.<fn>
dart:async 2402:21                          Timer.run.<fn>
dart:async-patch 15:15                      Timer.Timer.<fn>

Let’s take this opportunity to jump into some code. The Trace class is how the package represents a full stack trace. It can be parsed from any supported platform’s stack trace using new Trace.parse()stack_trace knows how to parse its own format, the Dart VM’s, and that of every browser Dart supports. If you know the platform you’re parsing for, you can also use platform-specific constructors like new Trace.parseFirefox(). Once you have a Trace, all you need to do is call toString() to get the nice, clean stack format.

import "package:stack_trace/stack_trace.dart";

void main() {
  // Throw an error so we have a stack trace to work with.
  try {
    throw '';
  } catch (error, stackTrace) {
    // Parse the trace from the StackTrace object. This works on the VM and on
    // any supported browser.
    var trace = new Trace.parse(stackTrace);

    // Print the stack trace in its pretty format.
    print(trace);
  }
}

Once you have a Trace, there’s even more you can do to clean it up. Most of the time, users only care about the internal workings of packages that they’re actually responsible for—they don’t care that List.reduce() called IterableMixinWorkaround.reduce(), they just care that somehow dart:core got from List.reduce() to their callback.

To address this, stack_trace has a notion of “folding” a Trace. This removes most of the stack frames that are considered irrelevant. It leaves only the ones immediately after relevant frames, since the user does care that their function called List.reduce() even if they don’t care what that called. Call Trace.foldFrames() to return a folded Trace, passing in a predicate that returns whether or not a line is relevant:

// Fold all the core library frames and frames from the test package.
trace = trace.foldFrames((frame) => frame.isCore || frame.package == 'test');

There’s also a built-in Trace.terse getter that folds together frames from the core libraries and the stack_trace package itself. It also does some bonus cleanup: it’ll clean up core library frames by getting rid of individual library names and line numbers, and it’ll remove the initial chunk of core frames that represent internal event loop junk. You can also get those bonuses by passing terse: true to foldFrames().

Here’s the terse version of the stack trace above:

dart:core                                   Object.noSuchMethod
pkg/stack_trace/lib/src/trace.dart 47:21    Trace.terse.<fn>
dart:core                                   List.reduce
pkg/stack_trace/lib/src/trace.dart 40:35    Trace.terse
pkg/stack_trace/lib/stack_trace.dart 24:28  format
test.dart 21:29                             main.<fn>

Gorgeous! Removing all of that extra text lets the user focus only on their code and the code it calls.

Parsing and Manipulating Traces

There’s more to stack traces than just printing them out nicely, though. A Trace, and the Frames it contains, makes it possible to inspect all the data encoded in a stack trace. You can get at all the information that’s directly included in the original trace via getters like Frame.uri, Frame.member, Frame.line, and Frame.column.

There are also derived getters that build in some logic for extracting common information. Frame.package, for example, returns the name of the package for package: URI frames, and returns null for any other frame. Frame.isCore is true for core library frames. Frame.library is a more human-readable version of Frame.uri, and Frame.location is a human-readable combination of the URI, line, and column.

You can also create your own stack traces from scratch using the new Frame() and new Trace() constructors. This ability is put to good use by the source_map_stack_trace package, which uses a source map produced by dart2js to convert a browser’s stack trace into one that has Dart files and line numbers. Here’s an excerpt of how it uses stack_trace:

return new Trace(trace.frames.map((frame) {
  // ...

  return new Frame(
      Uri.parse(sourceUrl),
      span.start.line + 1,
      span.start.column + 1,
      minified
          ? (span.isIdentifier ? span.text : frame.member)
          : _prettifyMember(frame.member));
}).where((frame) => frame != null));

Asynchronous Stack Chains

Writing asynchronous code is tough. Lots of stuff is happening in disconnected bursts without strong guarantees about ordering, and it can be really hard to figure out what’s happening where and why. You may have to think in four dimensions to understand your program flow in its entirely, but at least the stack_trace package can help you out with the Chain class.

Whether you’re running on the Dart VM or on a browser, the implementation’s notion of a stack isn’t all that useful in heavily-asynchronous code. It just includes frames from somewhere inside the DOM or dart:async to your callback, without any further context. In order to know what’s really going on, you’d need to know what the stack was when the callback was registered. And if it was registered in another callback, you’d need the stack before that, and so on.

That’s what a stack chain describes: a list of stack traces that, together, shows how you got from main() to wherever the stack was captured. And as you’d expect, a Chain object simply contains a list of Traces. The real magic is in how you get these chains: The Chain.capture() static method.

import 'package:stack_trace/stack_trace.dart';

void main() {
  Chain.capture(() {
    // Your program goes here.
  });
}

That’s the easiest way to use Chain.capture(): just wrap it around your entire program. Any errors that cause your program to exit will have their entire stack chains printed. Here’s an example of what that looks like (with Chain.terse called, of course):

test.dart 17:3       runAsync
test.dart 13:28      scheduleAsync.<fn>
===== asynchronous gap ===========================
dart:async           _Future.then
test.dart 13:12      scheduleAsync
test.dart 7:18       main.<fn>
package:stack_trace  Chain.capture
test.dart 6:16       main

That “asynchronous gap” is where one stack trace ends and the next begins. Here’s how you read it: main() called a bunch of stuff that eventually terminated in a call to Future.then(). Then once that future fired, it ran the callback in scheduleAsync() that called runAsync(). If the future had fired synchronously, the stack would look the same, just without the gap.

I called this “magic” earlier, but it’s actually possible to understand how it works. It’s based on Dart’s notion of zones, which you can read about in detail elsewhere. For our purposes, you just need to know that a custom zone lets us add extra behavior to async stuff, and that it’s active in its callback, any callbacks its callback registers, and so forth. Let’s call the zone Chain.capture() creates the capture zone.

The capture zone keeps track of a tree of stack traces. Each callback is associated with a particular node on that tree, and when that callback runs, the zone remembers its node. If the current callback registers a new callback, a new node—containing the current stack trace—is added as a child of the current node and associated with the new callback. And when an error occurs, we just add its stack trace to the current node’s stack chain to get a Chain object, which we associate with the trace that produced it.

This scheme lets us provide two important APIs. Because we have a chain associated with every trace, we can get the chain for a specific trace with new Chain.forTrace() constructor. This returns a full chain for any stack trace created in the capture zone, and for consistency it will even wrap stack traces in a Chain outside the zone (although it won’t provide any extra information).

Also, because we always know the current node in the tree of traces, we can get the current stack chain with new Chain.current(). This is useful for simple tasks like printing the current chain while debugging, but it also lets packages like scheduled_test store stack chains for use in future debugging. To suit the latter use case, you can pass in a number of frames to cut off from the bottom so users don’t see the library’s capturing function in their nice pristine stacks.

Since the stack chain capture zone stores a node for every registered callback, you might be thinking that it’s pretty performance-intensive. In practice, it’s less of a drag than it seems like it would be—partly because it’s set up so that nodes that can’t possibly be used are garbage-collected—but it’s definitely not something you want to leave on in production. That’s why Chain.capture() has a when parameter: just pass in false when you’re running in production mode, and all capturing is disabled.

Trace Your Stacks

Whether you go off and build a powerful stack trace processor or just start wrapping your main()s in Chain.capture(), you should be using stack_trace. No matter how good of a programmer someone might be, errors do happen, and stack traces are the most straightforward tool we have for understanding what’s going on. You’ll have to work with them one way or another—do yourself a favor and do it in comfort and style.

Monday, November 30, 2015

How Google Uses Angular 2 with Dart


Google is always busy building new web apps, and recently a lot of that development is using Angular 2 for Dart. Just last week, the Google Fiber team launched Google’s first production Angular 2 app, written in Dart. Take a look at their just-launched Angular 2 for Dart app:



The Fiber team has lots of company within Google—Angular 2 for Dart is being quickly adopted by many other Google teams.

Dart is an open-source development platform from Google that makes web and mobile development more productive. The Dart language is a concise yet expressive language that’s familiar to developers coming from Java, C#, ActionScript, and JavaScript. The Dart platform gives you an instant edit-debug cycle, great core libraries, a canonical package manager, a powerful semantic analyzer, and IDE support via IntelliJ IDEA and WebStorm. You can run Dart code compiled to JavaScript on all modern browsers.

We’ve worked closely with the Angular team as they’ve developed Angular and Angular 2, and we’ll be working together going forward. The Angular team’s initial experiences with Dart shaped many different aspects of Angular 2—for instance, Angular 2’s new change detection system and template compilation. Both of these features dramatically sped up Angular 2 for both JS and Dart with 3x faster startup and 2.5x faster rendering than Angular 1. This sort of speedup makes a big difference for our users, especially on the mobile web.

We started our collaboration with the Angular team by working on Google’s internal CRM application, Greentea. The Angular team built the initial version of Angular 1 for Dart as Greentea was built, providing lots of feedback to the Dart team. By working with the Greentea team and the Angular team, we were able to make large improvements in Dart compilation to JavaScript, Dart-JavaScript interop, and IDE support for Dart.

Once Greentea was up and running, the Angular team started working on Angular 2. We’re thrilled to have Dart as a first-class language for Angular 2. Beyond Google Fiber, teams that have publicly announced that they’re currently moving to Angular 2 for Dart include:

  • Greentea, Google’s internal CRM system.
  • Google AdWords, the largest advertising system at Google.

The Google Fiber team told us that they chose Angular 2 for Dart because “Our engineers were highly productive as their projects scaled to many engineers. Dart’s support for strong typing, its object-oriented nature, and its excellent IDE integration really worked for us.”

We're continuing to improve the tools for writing and deploying apps that use Angular 2 for Dart. For instance, we’re adding Dart Analyzer support for Angular 2 templates, which will allow navigation and refactoring across Angular 2 templates and Dart code. We’re also dramatically cutting the JS that’s output by the Dart2JS compiler—“Hello, World” is down to 82KB gzipped, and we’re continuing work to slice that further.

We hope that sharing our experience will convince you to give Angular 2 for Dart a try. Angular 2 for Dart is available for you to try as a Developer Preview now—take a spin through the Angular 2 Getting Started tour now!

Dan Grove and Kevin Moore for the Dart team

Wednesday, November 18, 2015

Dart 1.13 brings improved JavaScript interoperability and more

Dart 1.13 is now available. With this release, you can more easily access JavaScript APIs from Dart code. We've also improved secure networking on the server.

Easier JavaScript interoperability


Dart 1.13 provides a new syntax for creating Dart API facades for existing JavaScript libraries. Facades have the benefits you expect from a Dart library: errors, warnings, and code navigation. They also provide the Dart-to-JavaScript compiler the structure needed to provide interoperability with low code size and runtime cost.

Example facade for Chart.js library

Use the js package to create Dart APIs for your favorite JavaScript libraries. We have an example port of the Chart.js library if you'd like to see how to use these new features in your code.

Graph generated with Chart.js and Dart
We're actively working on tools to generate JS facades from other typed-JavaScript implementations. To track progress, subscribe to this GitHub issue.

Improved secure networking


As we mentioned in September, Dart has transitioned to BoringSSL – a streamlined, Google-maintained implementation of OpenSSL. As part of this change we have updated related APIs to use a standard PEM file for certificates and keys. Note: Existing code that uses TLS/SSH on the Dart VM will have to be updated to use the new APIs, as described in TLS/SSL with Dart.

And more...


The SDK changelog has details about all of the updates in Dart 1.13. Get it now.

Wednesday, October 21, 2015

Highlights from the TC52 meeting on September 21, 2015

Last month we had the 9th TC52 meeting where we discussed a number of additional language features for Dart. Below is a list of the main features discussed - most noticeably the lifting of restrictions on mixins.


DEP update
There is a rich supply of Dart Enhancement Proposals under consideration, and we spent some time discussing the underlying ideas and motivations for several DEPs.

Language extensions and modifications under consideration
Several language extensions and modifications are currently being considered in more concrete terms.

  • Generic methods
    This is the most complicated DEP, in particular with respect to reification of the type arguments and other typing issues. Since the previous meeting work has been conducted on prototyping features relating to this DEP, and the impact on libraries has been explored. Among the big concerns are compatibility and implementation efforts. This might not materialize before 2016.
  • Constant context, constant functions
    One DEP is concerned with constant contexts. They would enable many occurrences of the keyword const to be omitted, based on being part of a larger const expression. An upcoming DEP proposes constant functions, i.e., functions that map constants to constants. Concern was expressed that constants are already rather complex. In general, we wish to keep things more stable than they have been recently.
  • Import configuration
    There are many proposals for how to express configurable imports, in particular with respect to the level of static checking, and the support for library interfaces. These DEPs will not be finalized in this round.
  • Metaclasses
    This DEP proposes that Type objects should have more behavior. Not much has happened in this area since the previous meeting.
  • Relaxing restrictions on Mixins
    Three restrictions on mixin have been in place since their introduction in Dart. A DEP proposal for lifting two of them is available. An implementation of both features is largely ready. In particular, it has been implemented in the virtual machine dart, but work is still ongoing for dart2js.
    The situation is as follows: There was an original proposal encompassing a rather complete feature set. This proposal was restricted, in order to make it more implementable, such that mixins must inherit from Object; mixins cannot contain super calls; and mixins cannot define constructors. We are lifting the first two of these restrictions, and keeping the third one in place at this point.
  • Null-aware operators
    This feature has been adopted into the specification and implemented. It has been very well received.
  • Generalized tear-offs
    This feature has been adopted and mostly implemented. There are still a number of known bugs.
  • Package specification
    The language specification will state that this is an implementation dependent area. The use of a file named .packages to support the semantics of ‘package:’ imports has been implemented, and the detailed approach requires a little more flexibility than the existing specification allows. In general, we do not want to constantly adjust the specification to fit the needs of tools, which means that we will add flexibility rather than detail in these areas.
  • Function
    The type Function has been used in practice as a catch-all type for functions, providing the information that a given object is intended to be used as a function, but omitting information about the number, kinds, and names of arguments as well as their types, and about the return type. Following the current language specification, Function does not have a call method, so a conforming tool (compiler, analyzer, etc.) must warn at every invocation. However, the tools have not been doing that so far. The specification will be adjusted to make Function a special case, such that the omission of these warning will be the correct behavior. We will probably treat Function as a ‘magic type’ that supports call with all shapes of argument list, and returns a value of type dynamic.
  • Function subtyping
    The proposal makes function subtyping covariant in the return type, which is required in order to support message safety and in order to make refactoring safer. We discussed whether or not this topic should be considered to be part of the DEP on generic methods.
  • Library name conflict warnings
    The issue is that many programmers leave out the library directive, which means that they give their libraries the empty name; with two such libraries there is a conflict because they have the same name. The slides suggest that the solution to the problem of conflicting library names is to give them a name based on the URI. However, the decision was actually to treat libraries whose name is the empty string specially, and not give a warning in that case.
  • ‘noSuchMethod’ type checking
    We also discussed the liberalized rule for type checking classes other than Object with an implementation of noSuchMethod(). If a class implements an interface but lacks one of the methods of the interface, or if a concrete class has an abstract method, this usually causes a warning. An exception to this rule is when the class declares a noSuchMethod(). We propose to liberalize this so that an inherited noSuchMethod() suffices (except the one from Object of course).
  • Assert messages
    A new DEP is upcoming, on adding a string message to assert.

The committee unanimously approved the proposal for a 4th edition of the Ecma-408 standard - including the lifting of mixin restrictions. We're now awaiting approval from the Ecma General Assembly in December, 2015. The next TC52 meeting will be held on January 13, 2016.