Fixing JavaScript Date – Web Compatibility and Reality

In my previous post, I discussed things that could be fixed in JavaScript’s date implementation – if we wanted to. In this post, I’ll discuss things that can’t be fixed – no matter how much we want them to be.

No Semantic Versioning Here!

Most developers are familiar with Semantic versioning, or semver. In semver, we have the idea of three kinds of release:

  • Patch – releases that fix bugs only
  • Minor – releases that add small features
  • Major – releases that break things, so you have to change code

Semver is amazing. It clearly describes what versions we can safely upgrade to, when we should look for new features, and when we should expect to have to change code that is broken. It enables awesomeness. Unfortunately, it is not a reality for JavaScript.

A blessing and curse of JavaScript is that it is currently the world’s most popular programming language. Because of the sheer number of users, it’s safe to assume that anything that can be expressed with the language, has been expressed with the language, even if the code is what any sane person would consider terrible.

In addition, terrible code from 1998 is still being served to browsers everywhere, and there aren’t enough devs in the world to go update all of that code.

This results in two very important concepts that members of TC39 must constantly keep in mind:

  1. Web Compatibility – No change made to ECMAScript can be incompatible with the existing behavior of ECMAScript
  2. Web Reality – If code currently behaves a certain way, future versions of the spec should continue to have it behave that way – even if the behavior present is not described in the spec.

These concepts can really be summed up with the words “Don’t break the web!”. This creates a reality where there can be no such thing as a semver ‘major release’ in JavaScript. This idea drives every moment of the TC39 process, and has resulted in some unpopular but necessary compromises in the spec.

Mutability – a Web Compatibility Problem

I am a big fan of Domain Driven Design by Erik Evans. In the DDD world, objects can be differentiated as Entities which change over time and are tracked by their ID, and Value Types which are defined by their properties.  Under this definition, a DateTime is a value type. If any property of the date changes (for instance, the month changes from January to Feburary), the date is certainly a different date. Currently though, JavaScript doesn’t really work this way. Consider the following:

var a = new Date();
a.toISOString(); //"2017-04-05T05:57:53.350Z";
a.setMonth(11);
a.toISOString();//"2017-12-05T05:57:53.350Z";

As you can see, the value of object a changes. Yet, April and December are certainly different months, and these are certainly different dates. This kind of behavior sets people up for nasty bugs down the road. For instance, the following code will not behave as expected:

function addOneWeek(myDate) {
    myDate.setDate(myDate.getDate() + 7);
    return myDate;
}

var today = new Date();
var oneWeekFromNow = addOneWeek(today);

console.log(`today is ${today.toLocaleString()}, and one week from today will be ${oneWeekFromNow.toLocaleString()}`);
//today is 4/16/2017, 10:58:10 AM, and one week from today will be 4/16/2017, 10:58:10 AM

 

WOAH! This is no good. A better, and less bug prone behavior, would be to have the setters on the Date object return a new instance of the date – or for dates to be immutable. Then, the above code could be refactored to this common sense code:

function addOneWeek(myDate) {
    return myDate.setDate(myDate.getDate() + 7);
}
var today = new Date();
var oneWeekFromNow = addOneWeek(today);

console.log('today is ${today.toLocaleString()}, and one week from today will be ${oneWeekFromNow.toLocaleString()}');
//today is 4/09/2017, 10:58:10 AM, and one week from today will be 4/16/2017, 10:58:10 AM

Unfortunately, this is not to be because of a Web Compatibility issue. In short, if we were to make this change, tons and tons of code that relies on date being mutable (including the entire Moment.js library, BTW) would be broken.

Broken Parser – A Web Reality Issue

The ECMA262 standard currently describes very few rules for parsing date strings. These few excerpts are of particular interest:

ECMAScript defines a string interchange format for date-times based upon a simplification of the ISO 8601 Extended Format. The format is as follows: YYYY-MM-DDTHH:mm:ss.sssZ

This quote makes perfect sense. It states that JavaScript uses ISO8601 format as it’s main date interchange format. Since this is the most common interchange format for dates in modern computing, this is a great start! The standard then describes ISO8601 format options briefly:

This format includes date-only forms:

YYYY
YYYY-MM
YYYY-MM-DD

It also includes “date-time” forms that consist of one of the above date-only forms immediately followed by one of the following time forms with an optional time zone offset appended:

THH:mm
THH:mm:ss
THH:mm:ss.sss

So basically, you can have date only ISO8601 formats, and combined date-time formats. All good. But then a few lines further down, you get this wonderful quote:

When the time zone offset is absent, date-only forms are interpreted as a UTC time and date-time forms are interpreted as a local time.

In practice, what does this mean? Consider the following code:

new Date('2017-04-08').toISOString()
//"2017-04-08T00:00:00.000Z"
new Date('2017-04-08T08:30').toISOString() 
//"2017-04-08T15:30:00.000Z"

Basically, when I didn’t specify a time, the value was interpreted as a UTC value, but when I did specify a time it was interpreted as local. This is all… a bit mad. The next question one would ask is “is this some oddity of the ISO8601 specification?” But in fact that spec dictates that absent an offset, the time should be interpreted as local – meaning that under ISO8601 both values above should have been read as local time.

What happened here was a weird ‘Web Reality’.

In the days of ES5, the specification read this way:

The value of an absent time zone offset is “Z”.

This is saying that absent an offset, the time zone should be interpreted as UTC – the exact opposite of what the ISO8601 spec says. TC39 realized their error, and in ES2015 they corrected to this:

If the time zone offset is absent, the date-time is interpreted as a local time.

This is how the spec should read, as it aligns with ISO8601’s standard, and how basically every other date time API works. So why was it changed? Because as browsers started shipping this change, they started getting tons of bug reports that times weren’t being interpreted as they had been before – which of course they weren’t. By the time TC39 was able to give this issue more attention, the ecosystem was stratified, with some browsers picking UTC, and others picking Local, and others still making unique code compromises. After much evaluation of user feedback about what was expected to happen, the committee settled on the text as it is today, because it was the ‘reality’ of how the web worked – even if it wasn’t correct by the definition of the standard, or even particularly logical.

Given the amount of pain changes to this part of the spec caused the greater community, it is the case that it can’t be changed. The ‘web reality’ of what people expect from Date’s current parser will not allow it.

What do Do?

These two things were the impetuous for Matt, Brian, and I to choose to introduce a new datetime handling object to JavaScript, giving us a clean slate to make the world right. Currently we call this object ‘temporal’. This proposal can be found here, but look for a future post discussing our choices.

Fixing JavaScript Date – Getting Started

I’ve been off the blog for a while, which has to do with a lot of things going on in my life. That said, I’m happy to report that I’m back with stories about a big project – fixing the date handing in the JavaScript programming language itself!

History

A little backtracking here. I met Brian Terlson not too long after moving up to Redmond Washington to work for Microsoft. Brian is Microsoft’s representative to TC39, and the editor of the ECMA262 specification. 

I found Brian as part of a twitter conversation between me, Brendan Eich, and Moment.js co-maintainer, and fellow Microsoft employee Matt Johnson about how bad the Date handling in JavaScript is.

In this conversation, Brendan gave us some great history on the current Date object in JavaScript.

It is now common knowledge that in 1995 Brendan was given only 10 days to write the JavaScript language and get it into Netscape. Date handling is a fundamental part of almost all programming languages, and JavaScript had to have it. That said, it’s a complex problem domain and there was a short timeline. Brendan, under orders to “make it like Java” copied the date object from the existing, infant, java.Util.Date date implementation. This implementation was frankly terrible. In fact, basically all of it’s methods were deprecated and replaced in the Java 1.1 release in 1997. Yet we’re still living with this API 20 years later in the JavaScript programming language.

We all decided it was time for a fix and Matt, Brian, and I sat down to work.

Problems

In our first meeting, we identified the basic problems with the current Date implementation.
Here they are, in order of relative disgust:

  1. No support for time zones other than the user’s local time and UTC
  2. Parser behavior so unreliable it is unusable
  3. Date object is mutable
  4. DST behavior is unpredictable
  5. Computation APIs are unwieldy
  6. No support for non-Gregorian calendars

Fixable Stuff

Some of the issues mentioned are fixable in the current implementation of date. The lack of time zone support could be mitigated by adding a new factory function to date – something like:

var zonedDate = Date.inZone(zoneIdentifier, dateString);

This technique could probably also be used to support non-Gregorian calendars:

var calendarDate = Date.withCalendar('Hijri', dateString);

In that same vein, we could add APIs to Date to make computations easier. Right now, for instance, there is no way to add or subtract time – one instead has to perform a get/set behavior. To add one week to a date looks something like this:

var myDate = new Date();
myDate.setDate(myDate.getDate() + 7);

It wouldn’t be that big of a deal though, to spec something like an addDays() method, to make this a little nicer:

var myDate = new Date();
myDate.addDays(7);

The unpredictable DST behavior is another thing that can be fixed, and actually constitutes a bug in the current ECMA262 spec. This bug will be fixed in ECMAScript 2018 by this pull request. I’m planning on a later post to discuss the details of this change, and how “bugs” in the ECMA262 spec are handled.

Web Reality and Web Compatibility

Now we can see that lots of issues with date are fixable. However, we are left with two things on our list that aren’t: mutability and parsing. The reasons that these things can’t be fixed relates to two concepts that TC39 has to deal with a lot – web compatibility and web reality. These are the single most difficult things that TC39 has to reckon with, and I’ll tackle them in my next post.

An Aside: Standards committees are not a part of my regular job description as a SRE for Azure. This is a project driven from a passion for better JavaScript, and the OSS community. It wouldn’t be possible without the joint help of the JS Foundation, who I represent at TC39, and my division, Microsoft Site Reliability Engineering, which is allowing me to put work time and energy into improving technology. Thanks for believing I can do this, and giving me the resources to move forward.

Immutability – It’s Happening!

I’m super excited to highlight this awesome RFC from Lucas proposing to bring his Frozen Moment immutability plugin into the Moment.js core package as an optional add-on.

This RFC proposes a new Moment namespace, moment.frozen that wrappers all Moment functions with code that will automatically clone the Moment before performing mutations, creating an immutable API.

Moving forward, it is our desire to make the Frozen Moment immutable API the preferred way of interacting with Moment.js, effectively causing the library to become immutable. By making a second API, we allow legacy products to continue to function as before.

This all started when I wrote a blog post about why Moment isn’t immutable yet about a month ago. With the amount of enthusiasm we saw from the community, we decided to move forward with making Lucas’ plugin official.

As of right now, I am reworking our build process to allow us to better package this plugin.

In the meantime, we need community feedback on a few things:

  • Should this be packaged as a separate plugin, or just live in the core library as a second API?
  •  What should this API be named? Frozen is a working name that probably will not be adopted. We have discussed “m“ and “mom“ as possible options.
  •  Should the immutable API be a second global instead of being attached to the Moment global?
  • Should we simply make the “moment“ namespace immutable, and have a “momentLegacy“ namespace that users can use to overwrite the definition of moment for back-comparability if desired?
  • How should plugins and other libraries that depend on Moment interact with this API?
  • If we do overwrite the “moment“ namespace with an immutable one, should we release a compatibility build under a 2.0 version number? This build would have the same underlying code, but a mutable moment namespace.

Moving forward, we will be looking for contributors in a few key areas:

  • Documentation rework (needs to handle multiple APIs)
  • Package management
  • Build

In particular, someone with a UX background who wanted to help us improve the design of the docs page to support multiple APIs would be greatly appreciated.

Happy commenting!

Moment.js – Stuff we ARE doing!

Yesterday I wrote a post about how Moment.js isn’t immutable, and how it really isn’t in the cards for development right now.

I think as a team, we are sometimes not great about communicating things to our community that we are doing, and that are moving the library forward. As such, I wanted to write a short post about stuff we do have going on.

Next Major Release

The next major release of Moment.js and Moment TimeZone will have a complete rework of how the system handles time zones internally. This pull request is the completed work on the moment side. Tim has yet to get to the Moment TimeZone side, but it’s coming.

From a functionality standpoint for the library, this doesn’t really add much, though we might get in some syntactic niceties we didn’t have before. Internally though, we see a drastic improvement in our code base. This will allow us a lot more flexibility with new features down the road.

We also hope to have all of Moment changed over to Babel and Rollup by next major release. Again, this doesn’t add much for functionality, but it set us up to start using more ES2015 features. As a team, we have to decide what features we want to bring into the code and when. This is still in discussion.

There is no date for this, but all of this code is well in progress.

Other Stuff on the Table

If any of these are particularly important to you, let me know!

Why Moment.js Isn’t Immutable Yet

This post reflects my opinions about immutability in Moment.js. I do not speak for our entire team.

About five months ago now, I joined the maintainers team of Moment.js. I actually didn’t have much OSS experience before joining the Moment team, so jumping into such a massively popular library has been quite the experience. I have learned a ton about JavaScript, dates, and even how standards bodies work. What I have learned the most about though, is what it means to be an OSS maintainer.

The first thing that I will tell you about any OSS maintainer is that we all want to make you, the user, happy. While we’re admittedly getting some benefit from a ‘personal brand’ standpoint, none of us on the Moment team have ever made a single dollar off of the library directly. We do it for the love of code, and for the love of our users.

Sometimes though, we end up in a position where we can’t deliver to our users in the way that we want to. One such example is changing Moment to have an immutable API.
This issue about making Moment an immutable library has been open since 2014, and in fact discussions of immutability of the library date back to 2012. There are hundreds of comments on the issue, and it gets a +1 type comment or thumbs up at least once a week. With the increase in popularity of React.js, the interest in this issue is going steadily up. We have a couple other big issues that also get +1’d a lot, but the interesting thing about immutability is that the people giving feedback on this issue are PASSIONATE about it. This is not a ‘nice to have’ for them. They believe it is an essential feature of the library that is missing.

Now, it is important to know that all of Moment’s maintainers agree that date and time types should be immutable. If we were sitting down and writing a new date and time library today, that is how we would make it. However, changing Moment to be an immutable API has some  very big logistical concerns for us.

In fact, changing Moment from mutable to immutable is almost like making a different library. If this change is done ‘in full’ and not as an add-on, it potentially creates a breaking change for practically every line of code that uses Moment.

But what does that really mean?

Moment.js has been around for five years, which is an eternity in the world of JavaScript libraries, and it isn’t really slowing down in usage. A quick analysis of some stats:

  • 4.5 million NPM downloads a month
  • 15,000 visits a day to Momentjs.com
  • Average of 75 stack overflow questions a month in the last year
  • 40 GitHub Issues for June 2016 (so far)
  • 21 Pull Requests for June 2016 (so far)

These stats are just for Moment.js core. Moment TimeZone and the docs add many more issues and pull requests.

We have no way of knowing how many people are using Moment. We can know the NPM download number, but I would be surprised if that represents more than half of our user base. Other package managers, CDNs, etc are adding many more.

What we do know is that because of those numbers our workload currently looks like this:

Iskren – Moment.js releases, pull request reviews, major code changes
Tim – All Moment TimeZone code and releases
Isaac – GitHub issue management, pull request reviews
Matt – Moment TimeZone GitHub issue management, Stack Overflow questions
Maggie – Documentation maintenance, Stack Overflow questions

All of us do a little bit of every category you see there, but this is generally how everything shakes out. The critical thing to notice though, is that very little of this time is writing actual code for the library – our fantastic community of contributors is really driving that. We’re just making sure it gets delivered to people.

The trick is, between the paying jobs, family, and stuff that just comes up in life, all of us are kept pretty busy.

We’re actually not concerned about the time it would take to write the code for a new major version of Moment that was immutable. If we agreed upon a time frame, a couple of us could probably set aside the rest of life’s concerns for a couple weeks and get this done.

What we are concerned about is having two forks to maintain. From the outside, it’s easy to say “use SemVer, end of life the other fork, and move on”. We have too many users for this. The bottom line is that we can make as many ‘end of life’ and ‘end of support’ announcements about 2.0 as we want, and we can set as many timelines as we please. It won’t change the fact that Moment 2.0 will be in hundreds of thousands of code bases, if not more, for years to come. One need only look to the prevalence of websites still using jQuery 1.* versions to know this is true.

What this means is that we will still be getting GitHub issues, bug reports, Stack Overflow questions and possibly even pull requests on the 2.0 fork for (potentially) years. And unlike jQuery , we don’t have a foundation backing us to help us have the time to answer these.

And as I said in the beginning, we want to make people happy. So we really don’t want to tell everyone ‘too bad, so sad, upgrade to the newest immutable version.’ Because we’re all coders at our day job. And we all know how difficult it can be to get ‘upgrade JavaScript library’ on a development schedule, especially when that library has made a change like moving to immutability, where potentially every line of your code that uses the library could be affected.

In the face of all that, with the mutability vs immutability issue, it all comes down to this one fact:

The mutable library does what people need it to do.

In writing this library, Tim (the author), Iskren (the current lead), and the rest of the community managed to improve millions of code bases. People use it every day. It solves their problems.

An immutable API would be better. But would it be so much better that we should tell all of the users we have out in the world that we can’t support their (working) code as it is today?

We haven’t ruled out immutability. Maybe it is worth it. It will remain in discussion, along with work-arounds like making this plugin an official plugin that ships with the library, which is frankly a more likely route. But this is why it hasn’t happened yet.

I would ask anybody who is rallying for immutability in Moment this question:

Do you want Moment to be immutable, or is Moment not the library you want?