Friday, August 26, 2016

Dart 1.19: Improved developer experiences

Dart 1.19 is now available. Get it now!

We closely collaborate with the Flutter team on providing the best developer experience for Dart for Mobile. This includes language changes that optimize the Flutter development experience. Dart code for Flutter apps commonly have long argument lists. Before 1.19, a trailing comma was not allowed after the last argument. This made it really tedious when removing the last argument, or when reordering the arguments. With 1.19 we introduce support for an optional trailing comma after the last argument of a call and the last parameter of a function declaration. This makes code editing much easier:

If you've struggled with package directory symlinks, you're in luck. You can now opt-out of creating a package directory and the associated symlinks scattered around your project directories. The flag to provide to pub get and pub upgrade is --no-package-dir. We've posted a detailed writeup of the changes if you'd like more information. Please test your code with this flag. We intend to make --no-package-dir the default behavior in an upcoming release.

See full changelog here.

Download the latest release, and let us know what you think!

So Long, Symlinks!

Long ago, back in the dawn of time, when nary a Dart file was to be found outside of the SDK repository, imports in Dart only supported file paths. As the language got older and an ecosystem began to grow around it, it quickly became clear that paths weren’t enough: Dart needed a way to import libraries that didn’t care about the location of the file doing the importing. The language team talked to the VM team and they came up with a solution. In addition to file: URIs, Dart would support a new package: scheme by looking for packages in a packages/ directory next to the entrypoint, wherever that happened to be.

This worked pretty well—well enough to form the foundation for the pub package manager, written by Bob and myself. But it also had flaws. A lot of symlinks had to be generated all over every package—the test package currently contains about four hundred. Not only was this messy, many tools ended up following the symlinks and corrupting the package cache. So the team came together again and came up with the idea of a package configuration file that lists available packages as well as the concrete URIs where they can be found.

Support for package config was quickly added to all Dart implementations. In fact, pub has been generating them since Dart 1.12. The only task remaining was to gracefully turn off pub’s support for generating the packages/ directory and all the symlinks that come with it… but that turned out to be easier said than done.

The thing is, for all its flaws, the packages/ directory was pretty easy to work with from Dart code. You just had to do some path manipulation with Platform.script to figure out where the packages/ directory was, and from there you could do whatever you wanted. It wasn’t clean, but it worked most of the time, and there was no way to do the same stuff with .packages.

Once again conversation ensued. APIs were proposed, rejected, redesigned, tweaked, implemented, rolled back, iterated on, landed, stabilized1, and verified. It took seven releases to get everything just right, but we did it, and we’re ready for you to try it out. And I’m here to tell you how.

Disabling The Packages Directory

The first step on your journey to package enlightenment is to abandon your attachment to worldly directories. Relinquish your need for the comfort of familiarity by passing --no-package-dir2 to pub get (or upgrade or downgrade). Pub won’t generate a packages/ directory, nor of course will it symlink to the directory that it doesn’t generate. If you already had a packages/ directory, pub will get rid of it as well as any symlinks pointing its way.

For most packages and applications, this is all you’ll need to do! The latest versions of core Dart tools like pub and test fully support running with a package config. As long as your code isn’t manually dealing with packages, you’re good to go. And if it is, keep reading!

Core Package APIs

The core libraries have a few primitive APIs for providing information about package resolution. They form the foundation of more advanced APIs implemented in packages, but they’re also straightforward enough to use on their own. They cover many of the most common use-cases for dealing with packages. It’s important to note, though, that they only work in the Dart VM; the JavaScript compilers only deal with packages at compile-time, so they have no way of reporting anything about them at runtime.

The most broadly-applicable library package API is Isolate.resolvePackageUri(). It converts a package: URI into a file:3 URL that can be easily converted to a path. This is the easiest way to find resources in your package, or even public resources in your dependencies.

It’s important to note that this API is asynchronous, though: it returns a Future<Uri> rather than just a Uri. This is because the VM loads package information lazily, and the load process is asynchronous. If you have any package: imports in your code, the information will already be loaded, but it’s not safe to assume that—even code that uses lots of different packages may be run from a snapshot every now and then.

The other important core APIs tell you where the package resolution comes from. Isolate.packageConfig returns the URL of the package config file used for the current isolate, and null if a package root is being used. Its twin, Isolate.packageRoot, returns the URL of the current isolate’s package root, and null if a package config is being used. These getters are useful for forwarding the current resolution scheme—for example, if you want to start a Dart process that uses the same resolution as the current isolate, you can use these to figure that out.

I should probably also mention the Platform.packageConfig and Platform.packageRoot getters as well, but I do not recommend using them. They return exactly the text passed on the command-line for the --packages or --package-root flag, respectively. This means they’re unreliable: if the package resolution information was auto-detected, they’ll return null. If the current Isolate uses a different resolution scheme than the main isolate, they could return something completely irrelevant. Avoid them.

The package_resolver Package

If you want to go beyond the basics in your package code, package_resolver is the package for you. It provides classes that encapsulate strategies for package resolution—using a package config, using a root, or using nothing at all. Where code used to just pass around a packageRoot string, it can now pass a PackageResolver that says how to resolve URIs.

You can construct a PackageResolver by passing a package config map to new PackageResolver.config(), or by passing a package root to new PackageResolver.root(). If all you have is the URI for a package config, you can also use PackageResolver.loadConfig(). But most of the time, all you need is information about the current isolate’s package resolution, which you can get from the PackageResolver.current getter.

The resolver supports more or less the same APIs that the Isolate does: resolveUri() is equivalent to Isolate.resolvePackageUri(), packageConfigUri to Isolate.packageUri, and packageRoot to Isolate.packageRoot. But it builds other useful methods on top of these as well.

The packagePath() method returns the root of a package on the local filesystem. It assumes that you’re using pub, since it needs to look above packages’ lib/ directories, but this works in practice for essentially all Dart code.

The packageUriFor() method is the reverse of Isolate.resolvePackageUri(): it converts a file: URL into a package: URI that refers to the same file, if such a URI exists. This is useful for reporting nice-looking messages; it’s much easier to read a short package: URI than a long absolute path.

The processArgument getter is useful when running Dart processes or compiling Dart to JS. It returns an argument that can be passed directly to the dart, dart2js, or dartdevc executables that will cause them to use the resolver’s resolution strategy. If you passed using a custom package config map, it’ll even manufacture a data: URI for you4.

It even exposes the package config map directly through the packageConfigMap getter. Note that since the isolate API doesn’t yet provide this information directly, this may require re-loading it from disk.

But I Want It Synchronous!

All the PackageResolver APIs are asynchronous because they might be implemented using the asynchronous Isolate APIs. But asynchrony can be annoying to work with, and sometimes it’s not necessary. That’s what the SyncPackageResolver class is for. It exposes pretty much the same API as PackageResolver, but with no futures to be found.

The catch is that if you want to get a SyncPackageResolver, all the information—the package config map or the package root URI—needs to be available up-front. So SyncPackageResolver.current and SyncPackageResolver.loadConfig() both return Future<SyncPackageResolver>, since they have to wait for Isolate API futures and/or load information from disk.

You can convert back and forth between synchronous and asynchronous package resolvers easily using the asSync and asAsync APIs. This makes it possible for packages to take whichever one they work best with without causing pain for their users. For example, source_map_stack_trace takes a SyncPackageResolver so it can work synchronously, but shelf_packages_handler takes a PackageResolver since serving handling requests is already inherently async.

Why Bother With The Package Root?

The package_resolver package goes to pains to work well with both package configs and package roots, and to make it easy for users to support both. This raises a question: if pub is moving away from generating a packages/ directory, what’s the point of supporting it?

First of all, there’ll be a transition period where both modes will be fairly common. Pub still generates the packages/ directory by default for now, and the language implementations prefer it if it exists. A smooth transition relies on code’s ability to be used in both old and new settings.

But even in the future when packages/ files aren’t generated by default, support for package roots will be sticking around, and it can still be useful. It may not be the best way to run code manually, but a lot of code isn’t run manually—Pub’s transformer support, for example, loads isolates over HTTP, where it serves virtual packages/ directories.

In addition to supporting current tools, we want leave the door open for future tools to innovate as-yet-undreamed-of ways to handle Dart code. So we choose flexibility and continue to support package roots.

On Into The Future

Once there’s been a release or two with the --no-packages-dir flag and users have a chance to make sure their packages work with only a package config, we’ll disable the flag by default. You’ll need to pass --packages-dir explicitly if you want it generated. A while after that, we’ll hide the flag and stop officially supporting it. We may get rid of it entirely one day, but there’s no hurry—it’s not that hard to support.

So go forth and update your packages! Use the core library APIs and the package_resolver package, or one of the helpful packages I didn’t have a chance to get to: resource provides an abstraction for loading resources, and shelf_packages_handler makes it easy for a Shelf server to support a package root. Or write your own! The underlying tools are powerful, and I’m excited to see what you can do with them.

  1. Well, mostly stabilized. There are still a few corners of the API that are still in the process of being thoroughly tested. Bear with us if you run into them, and maybe wait another release cycle before you depend too heavily on running without the packages/ directory.
  2. This flag is new in 1.19, but some readers may notice its similarity to the undocumented --no-package-symlinks that pub used to support. --no-package-symlinks was added ages ago to let the Dart analyzer team experiment with their package config support, and we decided to tweak it a bit when making it public.
  3. Since the Dart VM supports loading packages over HTTP, it’s theoretically possible for it to return an http: URL. This doesn’t happen that often in practice, but it is a possibility—for example, the test runner can load test suites over HTTP from a pub serve instance.
  4. Note that there are usually limits on how long command-line arguments can be, so this may not work if the package config is too big.

Thursday, July 28, 2016

Dart 1.18: Laying foundations

Dart 1.18 is now available. Go get it!

The team has been focused on implementation details over the last six weeks. The API changes to the SDK are very light – see the CHANGELOG – but we have been working hard laying the foundation for a number of important projects.

Download the latest release. Let us know what you think!

Wednesday, July 20, 2016

AngularDart is going all Dart

Until now, the multiple language flavors of Angular 2 were written as TypeScript source, and then automatically compiled to both JavaScript and Dart. We're happy to announce that we’re splitting the Angular 2 codebase into two flavors – a Dart version and a TypeScript/JavaScript version – and creating a dedicated AngularDart team.

This is amazing news for Dart developers because:

  • The framework will feel more like idiomatic Dart.
  • It will make use of Dart features that couldn't work with the TypeScript flavor.
  • It will be faster.

This is equally great news for our TypeScript and JavaScript developers, by the way. Cleaner API, performance gains, easier contributions. Read more on the Angular blog.

Angular 2 for Dart is used by many teams at Google. Most famously by the AdWords team, but many other Google teams build large, mobile-friendly web apps. Some of the top requests from these teams were: make the API feel like Dart, provide a faster edit-refresh cycle, and improve application performance.

That’s exactly what we aim to do. We believe we can significantly improve both the performance and usability of AngularDart. For example, in the 2 weeks since we started work on the purely Dart version, we were already able to unleash strong mode on the code and were able to significantly improve the code quality (fixed 1000+ warnings).

One more thing

We're also happy to announce plans to release our library of Material Design components for Angular 2 Dart. These components are built purely in Dart, and they’re used in production Google apps. Watch for updates.

Here’s just one of the many Angular 2 Dart components we plan to release:

Dart was designed "batteries included" – it’s not just a programming language, but also a set of stable libraries, solid tools, a great framework — and soon, a repository of battle-tested UI widgets.

The components aren’t ready to be publicly released yet, but if you were thinking about learning Dart, now is a good time to start. By the time you’re ready to build production apps, you’ll have the building blocks at your disposal.

Dig into AngularDart

angular2 2.0.0-beta.18 is now available on the pub site. You can look at the source, file issues and create pull requests at dart-lang/angular2. If you already use AngularDart in a project, you can use pub upgrade now to get the latest version. Please join the angular2 Dart group to ask general questions.

Wednesday, July 13, 2016

Changes at

Today we simplified, making it reflect the current state of the project a little bit better.

We have for the fundamental Dart technologies—the language itself and the core libraries. And then we have separate websites for the different targets:
Some other changes we made:
  • Feature the pages that people visit most often.
  • Show the core goals of the project on the homepage.
  • Completely rework the information architecture, from domains down to individual pages and sections.
  • Set up for hosting event-related micro-sites.
  • Reimplement the sites to make maintenance easier.
More significant changes will come, but we needed to land these changes before going further.
If you notice that something's broken or could just be better, please let us know using the relevant issue tracker:

Wednesday, June 15, 2016

Unboxing Packages: path

I want to do something a little different with my blog post this week. When I’ve written about packages in the past, I’ve mostly done a high-level overview of their APIs and how they fit into the Dart ecosystem as a whole. But path is one of the very oldest packages in the ecosystem, and any Dart user who’s written any server-side or command-line apps is probably already familiar with the API.

So instead of a high-level overview, I want to do a deep dive. I want to talk about why we made the design decisions we made when writing path, and how we implemented our design effectively and efficiently. This post will be as much about how the package was constructed as it is about what the final product looks like.

Initial Design

It first became clear that Dart needed a solid solution for path manipulation when Bob Nystrom and I first started working on pub. Paths may seem simple on their face, but there’s a lot of hidden complexity when you need to make them work with all the edge case formats that can crop up across all the operating systems we support.

This became our first design constraint: make something that handles all the edge-cases. This is less obvious than it sounds: a lot of times, good design involves sacrificing some edge-case behavior to make the common case better, or even just simpler to implement. But we knew that path would be widely-used across the ecosystem, and we wanted users to be totally confident in it. If an application had to sanitize its paths before handing them off to us, we weren’t doing our job.

Another important early decision was to make the core API use top-level methods. We often look at other languages’ APIs for inspiration, but they were split on this point. Node uses top-level functions, whereas Java uses instance methods on a Path class. Ruby uses static methods for simple manipulation and a Pathname class for more complex ones. This didn’t provide clear guidance.

We decided to rely on a rule of thumb: only create a class when it’s the canonical representation of its data type¹. There were already a bunch of APIs, both in the core and in external code, that logically took paths and accepted only strings, not our hypothetical Path objects. Certainly everywhere the end user supplied a path, that path would be made available to the program as a string.

So we decided to go with the flow of the existing APIs and continue representing paths as strings. All the path manipulation APIs now take strings and return strings, and the world is simpler for it.

We chose functions for the package based on a combination of our own needs and APIs that were common among other languages’ path manipulation suites. Some of them, like join() and relative(), were pretty obvious. Others like rootPrefix() only became apparent because they filled holes in actual code. And a few, like prettyUri(), only got added well after the package was released.

The Quest for Correctness

We wanted to make our users confident in the correctness of path’s logic, which meant we had to be confident ourselves first. To do this, we wrote tests. Lots and lots of tests. Today, the package has 2.5 times more lines of test code than implementation code, and that’s how we like it.

Writing tests isn’t trivial, though. We had to be careful to include all the cases that came up in practice. This meant that, for every function where they were relevant, we tested combinations of:

  • Directory paths that did or did not end in separators.
  • Paths with zero, one, or two extensions.
  • Paths with multiple separators in a row.
  • Paths containing, or entirely composed of, the directory traversal operators . and ...
  • Absolute and relative paths.
  • Different formats of the current working directory.

We wrote those tests first for the Posix style of path, which are used by OS X and Linux. Then we ported them over to Windows paths², and added even more cases:

  • Windows supports both / and \ as separators, so we tested both and their combinations.
  • Not only does Windows support C:\-style path roots, it supports \\server\share\-style UNC paths as well.
  • You can also start a path with \ in Windows to indicate that it’s relative to the current working directory’s roots.

Determining the proper behavior for all of these involved looking up specifications online, manually testing path behavior on the command line, and a healthy amount of discussion about exactly the right way to handle edge-cases. These discussions led to our next round of design decisions.

Not all paths are valid. Sometimes they run afoul of an operating system’s rules for valid characters, and sometimes they just don’t make sense at all—consider the path /.., for example, or just an empty string. I initially advocated for throwing errors in these cases since in general failing fast is good, but we discussed options and Bob convinced me that path operations should never fail³.

While failing fast can make errors easier to track down, it also means that a defensive programmer has to be aware of the potential for failure anywhere it could occur. Path operations are frequently used in small utility methods that aren’t expected to fail, and most of the time their output is ultimately passed to IO operations which already need error handling.

So instead of throwing an error, the path operations just do the best they can on meaningless input. For most operations, /.. is considered the same as / and the empty path is considered the same as ., but we don’t work too hard to adhere to these definitions if it would get in the way of efficiently processing valid paths.

We also had to figure out what to do with paths that contained irrelevant characters, like foo//bar or foo/./bar, both of which are semantically identical to foo/bar. We ended up deciding to preserve the existing format as much as possible. The user would be able to explicitly call normalize() if they wanted clean paths, but otherwise they’d get something like what they passed in.

This decision made it easier to interoperate with other software that did, for whatever reason, care about the exact format of a path. For example, code using less-robust path manipulation logic might not be able to tell that foo/baz/qux was within foo/bar/../baz, so it’s useful for p.join("foo/bar/../baz", "qux") to return "foo/bar/../baz/qux".

Platforms and Customization

Paths are unusual in that their semantics are deeply platform-specific, but following those semantics mostly doesn’t actually require running the code on the platform in question. We wanted to take advantage of this to allow users to do path manipulations for platforms they weren’t using, but we also wanted to make the easy default use the current platform. This called for more design.

We came up with the idea of a Context object, which would take a style of path (Posix, Windows, or eventually URI) and the only OS-specific piece of data path manipulation used—the current directory path. Context had a set of methods that exactly mirrored the top-level functions in path. In fact, path’s functions just forward to a context!

We used contexts heavily in our own tests. They allowed us to run Windows path tests on Linux, for example, and to test operations like relative() without having to make any assumptions about the current directory.

While adding contexts, we also made a design decision that turned out to be a mistake in retrospect. We’d defined a Style enum for determining which platform style a context should use, which would have been fine if we hadn’t decided to make public the methods Context called on the style.

We had a vague notion that this would allow third-party packages to define custom styles, but no one ever did. Even if they’d wanted to, different path styles are so idiosynctatic that they probably couldn’t have encoded all the custom logic in the methods we provided. So instead we had a bunch of public API surface that was tightly coupled to the internal implementation of path manipulation.

Eventually the implementation needed tweaking in a way that affected the Style methods. We couldn’t change those methods, so instead we deprecated them and added an internal implementation of Style where we could add new methods privately. The lesson here is sometimes maximal extensibility isn’t worth the pain.

Making it Fast

When we first implemented path, we were primarily concerned with correctness and not speed. Our philosophy was (and is) to avoid optimizing packages until we have a clear idea of what parts are slowest and used most heavily. If the package started out correct and well-tested, we could be sure that any performance improvements later on preserved the necessary behavior.

But eventually the time came to make those changes. Users were doing path manipulations in performance-critical loops, and that meant it had to be fast. We set up a benchmark so we could track our progress, and used Observatory to see exactly what parts of our code were taking the most time. Then we called in Anders Johnsen, one of our resident performance experts who’s since moved on from Google, to see what he could do.

It turned out he could do a lot! Not only did our code get faster, we learned quite a bit about strategies for keeping it fast.

The first change was to avoid parsing the whole path. Our original code heavily used an internal ParsedPath class that eagerly parsed the entire path and exposed its components as fields. We still use this class for particularly complex functions, but for anything simple and performance-critical, we now deal with the string directly. This removes a lot of extra unnecessary work and allocations.

The second change was to stop using regular expressions. At the time, Dart’s regular expression engine was very slow. It’s since been dramatically improved, but explicit string operations still tend to involve a lot less overhead. We had been using regexps for very simple operations anyway, so switching away from them ended up being pretty straightforward.

Finally, we had to short-circuit early when possible. A lot of path operations were very complex in the worst case—they required a lot of logic and maybe even iteration over the whole path. But the worst case didn’t actually come up all that often, and it turned out to be pretty easy to detect when it didn’t. For example, Windows paths can have a lot of different roots, which makes finding the root difficult. But if the path starts with /, then it’s guaranteed to be a root-relative path, so the root is "/". These sorts of checks may seem nitty, but they helped a lot.

Coming Up For Air

I tried something new today, and I’m curious what you thought. Leave me feedback in the comments if you’d like to see more deep dives, or if you prefer my previous articles that gave a high-level overview of package APIs. For my part, I like writing both of them, so I’d be happy to continue doing a mix in the future.

Join me again in two weeks when I cover a very generalized package that’s only used in one place so far.

Natalie is still doing the writing here. I’m just hitting publish. – Kevin

1: If it were up to me, the File class’s methods would be top-level functions as well.

2: We also support url paths, but these were added later on.

3: There’s one exception to this rule: path calls Uri.base to figure out the working directory for operations like relative(), which is an IO operation and so could theoretically fail. In pratice, though, it basically never does.

Thursday, June 9, 2016

Dart 1.17: More performance improvements

Dart 1.17 is now available. Get it now!

We continued the work from 1.16 of working closely with some of our key users at Google to make sure Dart is even more productive for developers.

We've additionally optimized how our core tools deal with large applications, and have seen significant improvement over the last two releases. Dartium is now much more stable. We have improved the speed of the Dart Analyzer by more than 200% on large codebases. Last, but not least, Dartium startup time on large applications at Google has improved by a factor of 4.

We also made a number of smaller changes to core SDK APIs, please refer to the SDK changelog.