Floobits News

Floobits Atom: Fixing Bugs (in Atom)

2016-08-02T18:49:23-07:00

Our Atom package has been improved significantly since it was originally announced. Several of these improvements were thanks to fixes in Atom.

Fixing Video Chat: Content Security Policy

When Atom 1.7 was released, users complained that video chat showed a black screen. Our plugin hadn’t changed, so I suspected something in Atom was responsible. I soon tracked it down to Content Security Policy headers. The new version of Atom used a new version of Electron, which was based on a newer version of Chromium. The newer Chromium had stricter CSP behavior, such that self no longer included blob:. I found the relevant code in Atom and submitted a pull request. It took a while for the Atom team to merge it, but the fix is now in Atom 1.9.

Fixing Crashes: Avoiding `NODE_ENV` Erasure

Another issue that had been plaguing our Atom package was random React-related crashes. None of the stack traces involved any Floobits code, so it took me a while to figure out the root cause. Try to find the common thread in all of these issues:

I went gallivanting through the React code and managed to figure it out. React was starting up in production mode, then later switching to debug mode. React only initializes certain debugging data structures if it starts up in debug mode. Starting in production and switching to debug mode caused it to attempt to access object properties that didn’t exist. Oops.

React chooses modes by checking process.env.NODE_ENV. If it’s set to “production”, it will run in production mode. Otherwise, it uses debug mode. Atom initializes NODE_ENV to “production” in src/initialize-application-window.coffee. Something must change it later.

I finally tracked it down to src/environment-helpers.js:

70 // Fix for #11302 because `process.env` on Windows is a magic object that offers case-insensitive
71 // environment variable matching. By always cloning to `process.env` we prevent breaking the
72 // underlying functionality.
73 function clone (to, from) {
74   for (var key in to) {
75     delete to[key]
76   }
77 
78   Object.assign(to, from)
79 }

This clone() function is called when running atom from the command line. If Atom is already running, it replaces the existing environment with the new one, which usually lacks NODE_ENV. Oops.

I created an issue describing the problem, soon followed by a pull request to fix it.

A Side Note: Node’s `process.env`

While working on the NODE_ENV issue, I discovered some surprising behavior in Node’s process.env. Look at this REPL interaction:

> process.env.TEST = undefined
undefined
> process.env.TEST
'undefined'
> typeof process.env.TEST
'string'

That’s right: process.env stringifies any value you try to set. The only way to un-set an environment variable is to delete it. This isn’t as crazy as it first sounds. Environment variables can only be strings. Still, such behavior is not obvious. It’s documented, but let’s be honest: Developers only read docs when they don’t think they understand something. If process.env looks like an object and walks like and object and quacks like an object, developers will think it’s an object. A better solution would be to have explicit process.setenv() and process.getenv() functions.

Conclusion

Fixing these bugs taught me some new things about browsers, Atom, and Node.js. No matter how much experience one has with these software projects, there’s always plenty more to learn. To debug software is to be constantly reminded of one’s hubris.

Lastly: If you’ve tried Floobits for Atom before and found it lacking, I urge you to check it out again. In addition to these bug fixes, the UI has been revamped. It all makes for a much better experience. And if you do find issues, please report them. User feedback is incredibly helpful!

Lodash v4.0.0: Not All Bugs are Created `isEqual()`

2016-01-25T00:58:28-08:00

On January 13th (two weeks ago), Lodash v4.0 was released. After waiting a week (for others to shake out the bugs), I branched our repos, bumped the lodash dependency in them, and fixed compatibility issues. I tried lodash-migrate, but it wasn’t particularly useful. lodash-migrate works by running all lodash functions twice (once with the old version and once with the new), then logging if the results are different. That means any code with side effects will trigger a false positive or just break. Instead, I read the v4.0 changelog and reviewed every incompatible call in our codebase. (This wasn’t as hard as it sounds. There were only a few dozen.) After reviewing, testing, and manually poking around after deploying to staging, I deployed the new release to prod. Success!

…or maybe not. Soon after deploying, I noticed increased load on our back-end servers. When I ssh’d in to diagnose the issue, I saw that our back-end was replicating data like crazy. This didn’t endanger any information, but it was incredibly wasteful. Digging deeper, I noticed that if a single buffer in a workspace had changed, the entire contents of the workspace were copied in the next replication pass. Clearly, the lodash upgrade was responsible for this behavior, but how? I went back to my editor and looked at the code responsible:

187 ...
188 let local_buf = local_bufs[rbuf.id];
189 rbuf.deleted = !!rbuf.deleted;
190 if (!local_buf || local_buf.md5 !== rbuf.md5) {
191   log.debug("to_fetch: %s/%s", workspace_id, rbuf.id);
192   to_fetch.push(rbuf);
193   buf_cb();
194   return;
195 }
196 if (_.isEqual(local_buf, rbuf)) {
197   log.debug("Local copy of %s/%s matches remote. Not fetching.", workspace_id, rbuf.id);
198   buf_cb();
199   return;
200 }
201 log.log("Local copy of %s/%s differs from remote. Fetching.", workspace_id, rbuf.id);
202 log.debug("local: %s remote %s", _.keys(local_buf), _.keys(rbuf));
203 if (settings.log_data) {
204   log.debug("local:  %s", JSON.stringify(local_buf));
205   log.debug("remote: %s", JSON.stringify(rbuf));
206 }
207 to_fetch.push(rbuf);
208 ...

I could reproduce the issue on staging, so I enabled debug logging and data logging. I immediately saw that not only did rbuf and local_buf have the keys, they also had the same the data! What was going on?! The only thing that changed was lodash, so I opened a REPL and double-checked the behavior of _.isEqual():

> require("lodash").isEqual({"a": "a", "b": "b"}, {"a": "a", "b": "b"})
true
> require("lodash").isEqual({"a": "a", "b": "b"}, {"b": "b", "a": "a"})
false

WTF? I couldn’t believe it. I ran similar object comparisons a dozen times. I triple-checked the JS objects I was comparing. I had to be making a mistake. I thought, “This can’t be a problem with lodash. It’s so basic.” I double-checked the ECMA spec to make sure object keys aren’t ordered. (They’re not.) I installed lodash 3 and re-ran the offending line in a REPL:

> require("lodash").isEqual({"a": "a", "b": "b"}, {"b": "b", "a": "a"})
true

At this point, I was pretty sure this was a bug in lodash. I went to lodash’s GitHub project and prepared a bug report. Out of habit, I searched for existing issues. The result? An issue that had been closed 10 days earlier. A fix had been committed, but a new release hadn’t been tagged. :(To fix the issue in production, I applied the patch to lodash in each project’s node_modules, then deployed again. Not ideal, but at least the bug didn’t affect Floobits anymore.

Despite my comment on the issue, a new release still hasn’t been tagged. In my opinion, this is a critical bug. Object comparison is simply broken in lodash 4.0.0. I’m not sure why there’s been such a delay. While the code has been committed, a bug isn’t truly fixed until it’s deployed.

New Feature: Atom Chat

2015-12-20T15:49:40-08:00

A few months ago, we launched Floobits for Atom. Since then, we’ve been fixing bugs and adding features. Ever since the initial release, users have been asking for the ability to text chat. Well, it’s here.

Thank you, everyone, for being patient!

New Feature: Request Code Review

2015-12-02T12:22:49-08:00

If you use our web-based editor, you may have noticed a button in the upper-right with the label, “Request Code Review.”

The button does exactly what it says. If you click on it and give us a short description of what you want reviewed, we’ll get back to you with an estimate. We can even schedule a time to video chat or pair program with you.

Now, we’re bringing this feature to our editor plugins. If you use Atom, IntelliJ, or Sublime Text, you can request code reviews. If you’re connected to a workspace, this option is available in each editor’s command palette.

New Feature: Custom Highlight Colors

2015-11-09T13:55:36-08:00

If you use Floobits with IntelliJ, Atom, or our web editor, you’ve probably noticed that users’ selections have have different colors. You might also have noticed that these colors can’t be changed. If you get unlucky, your colleagues might have the same color. Then it’s hard to tell who is editing what.

Well, good news! We’ve updated our plugins and our site to allow each users to change their highlight color. If you’re not satisfied with your default color, just go to your user settings and select a new color from the drop-down. I’m a fan of lime.

Users occasionally asked us for this feature, but we didn’t deem it high priority until one organization was unlucky enough to have three of their devs stuck with the same color. Oops. At least it won’t be a problem anymore. Sorry.

Again, be sure to update your plugins! If you don’t, you’ll still see old the colors based on each username.

Lastly, we love feedback from users. If you have any Floobits-related comments or suggestions, don’t hesitate to contact us.

Bad Base64, a Not-so-Tricky Bug

2015-11-02T12:58:41-08:00

Last week, I explained how we found a bug that caused Node.js to crash when given invalid base64. That post was already rather long, so I didn’t explain how one of our back-end services was getting invalid base64. Today, I satisfy everyone’s curiosity.

The Key Clue

After deploying a patched Node.js, we added code to our services to detect and log invalid base64. Tailing the logs showed messages like this:

Invalid base64 for 5827/94 Screen Shot 2015-10-03 at 23.59.15.png:
�PNG\r\n\u001a\n\u0000\u0000\u0000\rIHDR\u0000\u0000\u0005�\u0000\u0000\u00028\b\u0006\u0000\u0000\u0000�wo�\u0000\u0000\f\u001aiCCPICC Profile\u0000\u0000H��W\u0007XS�\u0016�[R\b\t-\u0010\u0001)�7Az�\u001a:\bH\u0007\u001b!\t\u0010J\f��bG\u0016\u0015\\\u000b\*�(\*...

As soon as I saw that, I knew what the problem was. The line noise you see is a Node Buffer (in this case a PNG) that has been run through .toString(). To get base64, one needs to use buffer.toString("base64"). Somewhere in our codebase, we were missing an encoding parameter. But where, and why?

Our Service Architecture

To understand the issue, one also needs to know a little about our service architecture. Here is an extremely ornate diagram:

                         Front-end servers                  Master                   Slaves

                         +-------------+                    +---------+              +--------+
         +-------------> |httpd        |                    |colab    | +----------> |colab   | <-----+
         +-------------> |colabalancer | +----+----+------> |         | |\           |        |       |
         |               +-------------+     /    /         +---------+ | \          +--------+       |
         |                                  /    /                      |  \                          |
         |               +-------------+   /    /                       |   \        +--------+       |
Internet +-------------> |httpd        |  /    /                        |    ------> |colab   | <-----+
         +-------------> |colabalancer | +----/--------------------------\---------> |        |       |
         |               +-------------+     /                            \          +--------+       |
         |                                  /                              \                          |
         |               +-------------+   /                                \        +--------+       |
         +-------------> |httpd        |  /                                  ------> |colab   | <-----+
         +-------------> |colabalancer | +-----------------------------------------> |        |
                         +-------------+                                             +--------+

All clients connect to our front-end servers. If the request is HTTP(S) (port 80 or 443), Apache httpd handles it. If it’s our protocol (port 3448), our colabalancer service handles it. Also, if the HTTP(S) request is a websocket, httpd proxies it to colabalancer.

When a client sends its auth info (api key, secret, workspace, etc), the colabalancer asks colab master, “Which slave has workspace X?” When the master responds, the balancer connects to that slave and pipes data between it and the client. The master also makes sure that there are multiple up-to-date copies of each workspace. If a workspace’s replication count is low, it tells a slave to copy the workspace from another slave. That way if a server suffers a hardware failure, everyone’s data is safe.

Our Protocol

All Floobits clients use our Realtime Differential Synchronization Protocol (RDSP). Colab slaves use the same protocol to replicate data amongst each other.

You can read more about our protocol here, but the key point is that the current version of our protocol uses JSON, which isn’t binary safe. Originally, we didn’t think it would be necessary to synchronize binary files. After all, who edits binary files in Sublime Text, Vim, or Emacs? But users wanted it, so we implemented it. To get around JSON’s limitation, binary files are base64 encoded. Since earlier versions of our plugins didn’t understand this change to our protocol, we added a supported_encodings field to our auth frame. If that field doesn’t exist, we default to utf-8 as the only supported encoding.

Now guess which field was accidentally removed in our colab client code. That’s right, supported_encodings. :(

Piecing it Together

Here’s how the whole mess went down, step by step:

Colab master notices workspace 123 has a low replication count. Master picks candidates based on load and disk usage. It then tells slave01, “Fetch workspace 123 from slave00.”
slave01 connects to slave00. It sends no supported_encodings, so slave00 thinks, “Oh. This client only supports utf8.”
After authing and receiving room_info for workspace 123, slave01 sees that its copy of buffer 456 is out of date. It tells slave00, “Give me buffer 456”
Unfortunately, buffer 456 is binary. Slave00 dutifully sends the utf8-encoded (and now corrupted) buffer to 01.
Slave01 tries to decode the utf8-encoded buffer as if it were base64.
Node.js crashes due to a bug in its base64 decoder.

It may not sound too complicated when put that way, but troubleshooting this issue was very difficult. Because Node.js was crashing, most of our diagnostic and debugging tools were useless. There was no exception to catch, no error callback, no debug port we could attach to. As our previous post describes, it took a lot of thought, logging, and luck to track down the crash.

Conclusion

Looking back, there were many ways we could have avoided this bug: better tests, more in-depth code reviews, static typing, etc. I think the real lesson to learn is more specific: We shouldn’t have used our own protocol for back-end replication. While it is more efficient —and an interesting technical problem to solve— it wasn’t worth the extra complexity and potential bugs. We could have saved time and sanity building the same system with a battle-hardend system like Cassandra.

So if you’re ever thinking of making your own distributed data store, perhaps… reconsider.

Why is Node.js Crashing? A Deep Dive into a Tricky Bug

2015-10-26T09:59:01-07:00

In early September, we started seeing one of our backend services exit with the following error:

2015-09-10_23:28:11.75413 node: ../src/node_buffer.cc:226: v8::MaybeLocal<v8::Object> node::Buffer::New(v8::Isolate*, v8::Local<v8::String>, node::encoding): Assertion `(data) != (nullptr)' failed.

Typically, Node.js exits with a JavaScript stack trace. This error was at a lower level. Node.js was dying because of an assertion in its Buffer code. At worst, creating a new Buffer should throw an exception. This was a crash. This was a Node.js bug.

The first thing I did was look at the code containing the assertion error. This was in node_buffer.cc on line 225. I’ve included the entire Buffer::New() constructor below:

209 MaybeLocal<Object> New(Isolate* isolate,
210                        Local<String> string,
211                        enum encoding enc) {
212   EscapableHandleScope scope(isolate);
213 
214   size_t length = StringBytes::Size(isolate, string, enc);
215   char* data = static_cast<char*>(malloc(length));
216 
217   if (data == nullptr)
218     return Local<Object>();
219 
220   size_t actual = StringBytes::Write(isolate, data, length, string, enc);
221   CHECK(actual <= length);
222 
223   if (actual < length) {
224     data = static_cast<char*>(realloc(data, actual));
225     CHECK_NE(data, nullptr);
226   }
227 
228   Local<Object> buf;
229   if (New(isolate, data, actual).ToLocal(&buf))
230     return scope.Escape(buf);
231 
232   // Object failed to be created. Clean up resources.
233   free(data);
234   return Local<Object>();
235 }

Somehow, data was NULL. Thinking the realloc() on line 224 might have failed, I double-checked memory usage on the servers. They were not in danger of reaching any limits, and crashes didn’t seem to depend on memory usage. The service crashed while using 1GB of RAM just as often as it did while using 100MB. Dang. Not an easy fix. It was a slim hope anyways. Modern OSes don’t return NULL from malloc() and friends.^[1]

The next thing I did was man realloc, to try and figure out how it could return NULL. Except for an out-of-memory condition, it wasn’t possible. Even passing NULL to realloc() returned a usable chunk of memory:

If ptr is NULL, realloc() is identical to a call to malloc() for size bytes. If size is zero and ptr is not NULL, a new, minimum sized object is allocated and the original object is freed.

There was simply no way for Buffer::New(), executed sequentially, to fail in this way. Therefore (I reasoned), the bug must be another thread modifying shared state. Probably a hard-to-reproduce race condition. Ugh. Still, I needed to fix the issue. Desiring to know more, I tried to build a reproducible test case. Annoyingly, I could only trigger the crash in production and staging, and only when copying lots of data between instances of the service. Having a lot of other stuff to do, I mitigated the issue by reducing the peak rate at which the service copied data.

A month later, Matt finally got tired of seeing crash emails. We paired to try and find the underlying cause. When I was explaining the issue to Matt, I pointed out that realloc() never returns NULL. He double-checked the docs and disagreed. When he linked to the manpage describing realloc()’s behavior, it said:

If size was equal to 0, either NULL or a pointer suitable to be passed to free() is returned.

Wait, what?! It turned out that I had run man realloc on my mac, while he had googled “realloc” and clicked on the first result. That result described realloc() on Linux. The two behaved differently, and that difference was crucial. Again, here are the relevant lines from Buffer::New:

220 size_t actual = StringBytes::Write(isolate, data, length, string, enc);
221 CHECK(actual <= length);
222 
223 if (actual < length) {
224   data = static_cast<char*>(realloc(data, actual));
225   CHECK_NE(data, nullptr);
226 }

If realloc() returns null, actual must be 0. If actual is 0, StringBytes::Write() must have returned 0. But to get to line 220, length must be greater than 0. How could this happen? Well, length is only set on line 214:

214 size_t length = StringBytes::Size(isolate, string, enc);

So StringBytes::Size() thinks the buffer is a certain size, but StringBytes::Write() disagrees or fails in some way. To confirm this hypothesis, Matt and I added a printf() before the realloc():

220 size_t actual = StringBytes::Write(isolate, data, length, string, enc);
221 CHECK(actual <= length);
222 
223 if (actual < length) {
224   printf("actual: %u length: %u\n", actual, length);
225   data = static_cast<char*>(realloc(data, actual));
226   CHECK_NE(data, nullptr);
227 }

I deployed this custom Node.js build to staging, and soon saw crashes. The crashes were preceded by lines such as:

2015-10-17_19:22:45.89086 actual: 0 length: 11518

Yahtzee! We were on the right track. Now how could StringBytes::Write() return 0? We both suspected base64-encoded buffers. Delving into string_bytes.cc, we saw that StringBytes::Size() called base64_decoded_size(), which called base64_decoded_size_fast(), which basically returned length / 4 * 3.^[2] At no point did any of these methods check for valid base64 encoded data. They didn’t strip whitespace or invalid characters. They just multiplied by 0.75.

It was a different story for StringBytes::Write(). That function called base64_decode(), which called base64_decode_fast(), which could return early with invalid base64 data. In that case, base64_decode() falls back to calling base64_decode_slow(). Let’s take a look at that function, which starts at line 167 of string_bytes.cc:

167 template <typename TypeName>
168 size_t base64_decode_slow(char* dst, size_t dstlen,
169                           const TypeName* src, size_t srclen) {
170   uint8_t hi;
171   uint8_t lo;
172   size_t i = 0;
173   size_t k = 0;
174   for (;;) {
175 #define V(expr)                                             \
176     while (i < srclen) {                                    \
177       const uint8_t c = src[i];                             \
178       lo = unbase64\(c\);                                     \
179       i += 1;                                               \
180       if (lo < 64)                                          \
181         break;  /\* Legal character. \*/                      \
182       if (c == '=')                                         \
183         return k;                                           \
184     }                                                       \
185     expr;                                                   \
186     if (i >= srclen)                                        \
187       return k;                                             \
188     if (k >= dstlen)                                        \
189       return k;                                             \
190     hi = lo;
191     V(/* Nothing. */);
192     V(dst[k++] = ((hi & 0x3F) << 2) | ((lo & 0x30) >> 4));
193     V(dst[k++] = ((hi & 0x0F) << 4) | ((lo & 0x3C) >> 2));
194     V(dst[k++] = ((hi & 0x03) << 6) | ((lo & 0x3F) >> 0));
195 #undef V
196   }
197   UNREACHABLE();
198 }

This macro-fied code may be a little hard to follow, but the behavior we care about is straightforward. Look at lines 182 and 183. Any “=” in the data causes the function to return early. It doesn’t matter if src is a megabyte. If the first character is “=”, k is still zero when line 183 is hit. Once we figured that out, it wasn’t too hard to reproduce the issue in a line of JavaScript. Try this with Node.js (or io.js) from v3.0.0 to v4.2.1:

ggreer@lithium:~% node
> new Buffer("=" + new Array(10000).join("A"), "base64");
node: ../src/node_buffer.cc:225: v8::MaybeLocal<v8::Object> node::Buffer::New(v8::Isolate*, v8::Local<v8::String>, node::encoding): Assertion `(data) != (nullptr)' failed.
zsh: abort (core dumped)  node
ggreer@lithium:~%

Armed with a one-liner crash, I reported the issue and described how I thought it was breaking. It only took a day for Ben Noordhuis to fix the bug in master. Node.js v4.2.2 and v5.0.0 shipped with the fix. Mission accomplished!

Except… we didn’t answer some important questions: Where was this invalid base64 coming from? Why was our back-end service processing it? Those answers are in the next post.

Thanks to Matt Kaniaris, Ben Noordhuis, and the rest of the Node.js team for their help.

Even when a process asks for more memory than is available, modern OSes return a usable pointer. Only when the memory is accessed will the OS jump into action and free memory by killing processes.
Why length / 4 * 3 instead of length * 0.75 or (length * 3) / 4? This is C++, so the former requires a type conversion to float or double, followed by rounding. The latter could overflow if length is greater than SIZE_MAX / 3.

Developing Atom Packages, Part 2: So Much Potential, So Many Bugs

2015-10-14T08:58:38-07:00

Atom has improved drastically since its first release, but it’s still not perfect. While building our Atom package, we ran into quite a few bugs. Some have been fixed, but many still persist. Previously, I discussed Atom and its progress. That post addressed larger, broader issues. This post gets into specifics. Lots of specifics.

Yay, it’s a Browser. Oh no, it’s a Browser!

Atom is based on Chromium. While this does have the advantage of allowing for packages to be written with JavaScript and HTML, it also causes some unexpected problems. For example: In a browser, submitting a form causes a page reload. Atom behaves the same. Without a handler on the form that calls event.preventDefault(), Atom will reload the entire window, losing state and breaking many UI elements.

Another issue with Atom being a “sorta-browser” reveals itself when one tries to load remote content, such as a website. Iframes are the obvious choice, but many sites kill frames to prevent clickjacking. The only solution is to use the Chrome-specific webview tag. Unlike frames, webviews in Atom can’t use postMessage() to communicate with the parent window. Instead, the parent has to bind to the console-message event. This receives data from console.log() calls in the webview. See our web view template for an example. This solution worked for us, but it’s not a cure-all. If you can’t add console.log() calls to the pages loaded in the webview, there’s no way to get messages out.

If your package has any visible UI, you’re also likely to run into styling conflicts. All Atom packages can include CSS or LESS, and all of their styles are applied globally. In addition, Atom loads a subset of Bootstrap for its own styling. To improve load time, Atom’s Bootstrap is CSS, not LESS. That means you won’t be able to @import it for your own package’s use.

Packaging

Compared to other editors, Atom’s packaging system is amazing. Most editors don’t even come with a package manager. Instead, users are forced to install third-party managers such as Package Control or Vundle. Atom’s package manager is great for users and developers. Users can easily find, install, and update packages. Developers can specify dependencies using npm’s package.json format. Packages can even depend on other packages. For example, our Atom package depends on Term3. That allowed us to avoid copying a bunch of terminal-related code into our main package’s source tree. It also means user settings for Term3 are always consistent. With luck, we’ll be able to get our changes merged into Term2 and depend on it instead. If not for Atom’s package management, we would have a complete fork with polluted history. The prospect of merging would be all but hopeless.

Of course, like many parts of Atom, packaging does have a few rough edges. Packages can load other packages… most of the time. If a package uses activationCommands, you’re out of luck. There are a few undocumented workarounds, but these hacky solutions are likely to break in minor Atom updates. Though packages can declare other packages as service dependencies, and they can (sometimes) load them, the dependent services will not be automatically installed. Users have to manually install them.

The View System

Atom’s view system has been a long series of misadventures. Early on, Atom recommended Space Pen, which was little more than a wrapper around JQuery. Then, Atom moved to React. Because package writers coded their packages against Atom’s React, Atom is forced to stay on an old fork (multiple versions of React can’t run in the same context). It’s unclear what will happen to Atom’s React, but its future doesn’t seem promising. Atom devs have been slowly removing React from core parts of Atom. More recently, Atom has started to implement parts of their visible UI in special-purpose HTML tags. This doesn’t seem to be the end-game for the view system though. The discussion around this topic has yet to nail down a solution.

To summarize:

Atom View API	Status
Space Pen	Deprecated
React	Deprecated
Special DOM elements	Maybe deprecated soon?

It’s unclear what’s next for Atom’s view API, but hopefully the community settles on something.

The API for managing windows is also a mess. For example, atom.open({'pathsToOpen': ['~/code/example'], 'newWindow': false}); opens a new window, despite the fact that newWindow is false. Annoyingly, atom.open() can also close current windows. The exact circumstances in which this happens are (so far) unpredictable. This bug is bad enough that we’ve had to add a disclaimer in our Atom package.

Miscellaneous

Atom’s API and internals have changed significantly since its first release, but a few annoyances have persisted. A big one for us has been resize detection. It’s very useful to know when a pane was resized. The only way to do that right now is inefficient polling. (See the implementation here).

A few pernicious issues stem from faults in Atom’s architecture. Packages have no isolation, so it’s easy for them to step on each other’s toes. Styling conflicts are the most common version of this. Much of Atom itself is implemented as packages, so this lack of isolation means that misbehaving packages can hang or crash Atom. Fixing this is an immense task, as it requires completely changing the plugin architecture. Even if Atom goes down that road, there’s no perfect solution. As Chrome extensions show, isolating packages has its own disadvantages.

One thing that Atom has done a great job of is unbinding event handlers. Their disposable objects make cleanup of event handlers trivial. Instead of forcing developers to keep track of bound functions, disposables have a dispose method that unbinds the event handler. Disposables can also be collected into a CompositeDisposable. I wish more JavaScript codebases used disposables.

This post may seem critical of Atom, but that’s not the case. I took the time criticize Atom because I care about it, and I want it to be better. I think Atom is a fine editor with a ton of potential. It has come a long way in a short time. Hopefully, Atom developers will use this post to improve it even more.

Developing Atom Plugins, Part 1: On the Bleeding Edge

2015-10-12T22:16:22-07:00

Back in February of 2014, GitHub announced their new editor: Atom. We’ve followed Atom since it went public, and recently developed a Floobits plugin for it. What follows are our impressions from the experience.

The History of Atom in 30 Seconds

At the time of its release, Atom was slow, buggy, and lacked basic features. Many wrote it off it as cheap imitation of Sublime Text. In the subsequent 18 months, Atom has improved remarkably. The recent 1.0 release is a powerful, extensible editor suited for everyday use. Atom still isn’t as fast or as stable as Sublime Text, but it’s catching up quickly. Or, to put it more accurately: Sublime Text development has stagnated. Since the first release of Atom, Sublime Text 2 has had zero releases. Sublime Text 3 Beta has had three, one of which was a minor bug fix. Considering the difference in development speed, Atom will almost certainly improve faster than Sublime Text.

Not All Rainbows and Unicorns

That said, Atom’s journey hasn’t been entirely pleasant. Its development process is accurately described by Facebook’s slogan: “Move fast and break things.” Since its initial release, Atom’s API has changed drastically, breaking package compatibility multiple times. Even the license has changed. Most of these changes have been for the better, but keeping up with them requires significant time and effort. Hopefully, things will settle down now that Atom is 1.x.

It’s a Browser… Sort Of

Many of Atom’s detractors point out that it’s based on a browser: Chromium. While this does increase resource usage and startup time, there are significant advantages to building on top of Chromium:

Fewer cross-platform issues. Chromium already works well on OS X, Windows, and Linux.
Plugins are written using JavaScript, HTML, and CSS. Web developers can quickly learn to extend Atom.
Tracking down errors in Atom is much easier than other editors, thanks to Chromium’s debugger. Only IntelliJ has comparable self-debugging abilities.
Chromium has browser technologies such as WebRTC, which aren’t available in any other editor or IDE. Floobits for Atom uses WebRTC for video chat. It’s very nice.

Leveraging a browser has advantages and disadvantages, but I think GitHub made the right choice.

A Few Missing Pieces

The Atom devs have done a great job over the past 18 months, but a few conspicuous issues still linger. By far, the most glaring bug is binary safety. If you open a binary file with Atom and save it without making any changes, the file will be corrupted. The only other editor I know that does this is nano.

Another glaring omission is Atom’s lack of GitHub integration. Some extensions are GitHub-specific, but the editor itself has no GitHub features. Built-in GitHub authentication would be useful for many extensions. Instead of setting your GitHub API key/secret in each plugin, Atom could provide it (with appropriate prompting, of course).

Despite these shortcomings, I’ve found myself using Atom more and more. It’s become a very compelling alternative to Sublime Text.

For a more technical discussion of Atom’s current issues, read part 2.

IntelliJ IDEA Plugin Development and Java 6

2015-08-27T14:20:00-07:00

Based on my experience of the last couple of years writing editor plugins for Floobits, I can say that IntelliJ IDEA’s plugin system is phenomenal. Extending IntelliJ IDEA is particularly stellar. Unlike most editors, IntelliJ’s plugin API gives you access to everything. The IDE architecture is very well designed. The core is open source and easy to follow. And IntelliJ plugins have all of Java’s libraries available. It is quite amazing.

There is a lot to rave about, but one thing that isn’t great is Java versioning. IntelliJ IDEA is a cross-platform Java-based editor, and its plugins must share the JRE that the editor runs on. The version of Java you choose for writing your plugin must be compatible with your target audience’s installed Java. If the user does not have the correct version of Java, your plugin will cause a nasty error at start time and your plugin will be automatically disabled. There is no way to specify in the plugin configuration file (plugin.xml) which version of Java you support. That means there’s no way to prevent people from installing something that will explode on them. If a user encounters this error, there’s no easy way to explain what happened or what to do about it.

The solution recommended by JetBrains is to use Java 6 for plugin IntelliJ IDEA development. But Java 6 is old. It’s so old that Oracle no longer provides security updates. Compared to Java 8 (or even 7), Java 6 is terrible to work with. There are no lambdas. There are none of the new nio APIs to reduce the pain of working with file I/O and threads.

Most people are running at least Java 7 now, so I looked into why JetBrains recommends Java 6. It turns out that for OS X users, Java 6 is what most people have, if they even have Java at all. Apple used to develop their own version of Java for OS X, but they gave it up, and the last version they supported was Java 6. Apple still makes this old version available, and it still gets security updates, but Apple refers to it as ‘legacy’ and has no plans to support any new versions of Java. Oracle does supply newer versions of Java that are OS X compatible, but they have a severe problem: The font-rendering is atrocious on OS X. This is a huge problem for an IDE! Blurry, pixelated, aliased fonts are not what you want to stare at for hours at end. Oracle is looking to fix this for newer versions of Java, but for now we are stuck with Java 6.

Given the state of Java on OS X, JetBrains has done a great job. Recently, they forked OpenJDK, fixed the font-rendering on OS X, and built versions of each of their IntelliJ IDEA based editors that include this version of Java. Unfortunately, this download is not the default. Discovering it requires clicking through a link for “previous releases.” In my opinion, this version of IntelliJ IDEA should be the default version for all OS X users and the version without a JRE should be an alternate download. Sadly, even if this was the case, there would still be many Java 6 installs. For now, plugins must be written to match the lowest common denominator: the JRE for Java 6.

I asked about Java version detection on the JetBrains plugin development forum, and was told:

There is no possibility to mark a plugin in this way (and we don’t plan to provide one). If you want to use lambdas and new things, please consider writing your plugin in Kotlin, which does have lambdas and new things, but compiles to regular Java 6 bytecode.

Our IntelliJ plugin is already written in Java, but if we were to build it today, we’d probably use Kotlin. Kotlin is a pretty amazing programing language. It provides many modern programming language concepts, such as lambdas, null-pointer saftey, and first-class functions. It does this without making it difficult to use Java libraries and code. Though using Kotlin still wouldn’t give us access to the nio APIs, the situation would be better.

In order to bring Floobits to as many editors and platforms possible, we have to endure the pain that is editor plugin development. Whenever we build plugins, we run into issues that most developers would walk away from. We don’t have that option, so Java 6 it is. Having said that, IntelliJ IDEA actually has the best plugin development API out there. This post doesn’t mean to detract from that.

If you’re curious, you can see our IntelliJ plugin’s code on GitHub.

Sublime Text's Plugin API: It's Python... Sort Of

2015-08-17T15:37:18-07:00

“Sort of” is such a harmless thing to say… sort of. It’s just a filler. Sort of… it doesn’t really mean anything. But after certain things, sort of means everything. Like… after “I love you”… or “You’re going to live.”

— Demetri Martin

Sublime Text is my editor of choice. It’s powerful, flexible, and accessible. But as much as I like it, Sublime Text has one glaring problem: its plugin API is Python… sort of. That “sort of” is responsible for much frustration and annoyance.

For most devs most of the time, writing Sublime Text plugins is scarcely different from writing normal Python. Just import some modules, define your Sublime commands, and you’re golden. At Floobits, we learned the hard way that this isn’t always the case. Two years ago, after months of extensive development and testing, we finally judged our Sublime Text plugin ready. We released it to the world… only to suffer a deluge of complaints. Many on Windows and Linux couldn’t use our plugin. Merely activating our plugin raised exceptions on their systems.

This confused us. Though we develop on Macs, the code should work fine everywhere. Except for a few well-documented edge cases, Python behaves the same on Windows, OS X, and Linux.

Sadly, that’s not true for Sublime Text’s Python. The Python that ships with Sublime Text varies significantly in its capabilities. On some platforms and Sublime Text verisons, standard modules are broken. You can verify this yourself. Open Sublime Text 2 on Windows and run import select in the console:

Or if you’re on Linux, open Sublime Text 2 or 3, then try to import ssl^[1]:

For Floobits, these errors were deal-breakers. Our plugins need select() for asynchronous network I/O. Our network protocol always uses encryption, necessitating the ssl module. For our purposes, these modules had no alternatives or substitutes. At the time, this realization was extremely disheartening. We racked our brains for solutions. It took a while, but we had an insight: Sublime’s Python might lack ssl, but the operating system’s Python should be fine. If we could hand encryption/decryption work to the OS’s Python, we’d be in business. Once we knew what to do, our solution took only a week to code.

Our users were happy, but the experience left me disillusioned. I reported the import ssl issue almost two years ago, but it still hasn’t been fixed. At this point, I doubt it ever will be.

This may not error for users of Package Control, which can install its own ssl module.

Introducing Floobits Pro

2015-08-12T12:47:53-07:00

Today, we are proud to introduce Floobits Pro. Floobits Pro lets you to hire other developers to help you out on your project. Whether you just need another pair of eyes to double-check a function, or an expert to track down a race condition, we’ve got you covered.

Signing up is easy. Go to our sign up form, provide us with your billing information (you will not be charged until you complete a session), and tell us what you need. We’ll create a workspace where you can add files and collaborate.

If you already have a Floobits account and you want to get help in an existing workspace, just click on the “Floobits Pro” button in the upper-right of the web editor.

Once you request help, we’ll alert the Pros who in turn choose to work with you. We provide you with a description of the technology they are familiar with, their feedback from previous sessions, and give you the option to chat before you make your decision of who to hire.

After you pick a Pro, they’ll immediately join your workspace and be ready to help. You can use all the tools that make Floobits great: work in your native editor, share terminals, and video chat using WebRTC or Google Hangouts. If for some reason it doesn’t work out, don’t worry. Cancel the session in the first five minutes and you won’t pay anything.

One more thing: We’re giving all of our paying users an hour of free help. That’s right: If you are on a paid plan, we’ll help you with your code for free! Think of it as a thank-you for your support.

We are very excited about Pro and hope you’ll give it a try. If you have any comments or suggestions, don’t hesitate to contact us. If you are interested in becoming a Pro, apply here. If you are concerned about the future for Floobits “Classic”, don’t be. We will continue to develop and support our plugins for native pair programming.

New Chat Feature: Floobot

2015-03-29T22:06:28-07:00

If you chat in workspaces or our organization chat, you might notice that pasting URLs can cause a mysterious “Floobot” user to describe them. Don’t be startled, this is expected behavior. Similar to showing images in chat, Floobot helps you to get information about a URL without having to click on it.

Floobot does more than just parse <title> tags. If the link is to a GitHub repo, it will mention the number of forks and stargazers. If the link is a tweet, it will give a summary, along with the number of favorites and retweets. Floobot also describes YouTube videos and Hacker News submissions. Basically, if you paste a URL, Floobot tries its best to give a one-line summary of what’s there.

Here’s a screenshot of what Floobot typically looks like in org chat:

Currently, Floobot’s code is based on our open-source IRC Floobot. We plan to split out the site parsing code and turn it into a node module. That way, everyone can take advantage of our tool. In the mean time, enjoy the useful descriptions of URLs. And if you have any suggestions or bug reports, please email us or open an issue.

Update: Our site parsing code is now in an open-source module. Feel free to use Fleece in your own projects.

IntelliJ IDEA User Interface Updates

2015-03-24T16:25:54-07:00

We’ve listened to your feedback and made some big improvements to our IntelliJ IDEA plugin.

First, it’s easier to see where others’ cursors are. When someone else moves to a new line, you’ll see a small indicator with their Gravatar and username. The border of the indicator matches their highlight color.

Second, we’ve updated the list of people connected to the workspace. Everyone now gets a larger version of their Gravatar, along with a list of the clients they are connected with. Previously, each client was a separate item in the list. Now, we’ve grouped them by user.

Also, right-clicking on a user opens a context menu allowing you to follow, unfollow, or kick clients or users. In addition, you can edit other users’ permissions from this menu.

One more thing: Our changes aren’t just cosmetic. We’ve improved performance! Specifically, highlights are rendered significantly faster.

This update should work in the most recent version of IntelliJ IDEA and its forks: PHPStorm, RubyMine, PyCharm, WebStorm, Android Studio, CLion and AppCode.

If you notice any issues, or have any feedback about these changes, don’t hesitate to contact us at support@floobits.com. We worked hard to make sure the changes performed well while being unobtrusive, but we’re open to feedback. Our IntelliJ IDEA plugin has steadily increased in usage, and we’re excited to continue improving it. Your suggestions are a big part of that.

On Moving to io.js

2015-02-23T15:09:46-08:00

Last weekend, we migrated our production environment from Node.js to io.js.^[1] This move may seem premature, but we had several reasons for switching.

For starters, we were growing increasingly frustrated with Node’s stagnation. Node.js v0.10 came out in March of 2013, 10 months after v0.8’s release. Work on v0.12 began the same month. In January of 2014, Joyent’s TJ Fontaine claimed v0.12 was “imminent”. Yet v0.12 finally shipped in February of 2015, over a year later. Compared to previous releases, those two years of development didn’t bring many changes. Just look at API differences between v0.8-v0.10 and v0.10-v0.12. It’s sad that such a useful, popular project has run out of steam lately. Even Node’s latest release ships with old, unsupported versions of V8 (3.28.73) and libuv (1.0.2). io.js has current releases: V8 4.1.0.14 and libuv 1.4.0. These bring performance improvements, bug fixes, and many new features.

The decision to switch wasn’t driven solely by Node’s issues. Many of io.js’s features appealed to us. For example, io.js supports much of ECMAScript 6 by default, and its support is more extensive than Node’s --harmony flag. This may seem trivial, but it’s actually a big deal. ES6 transforms JavaScript into a more powerful, more forgiving, and generally more pleasant-to-use language. Template strings are a welcome bit of syntactic sugar. let and const help programmers avoid whole classes of mistakes. Generators and promises make it easier to translate one’s mental models into straightforward code. In short, ES6 has changed how we write server-side JavaScript more than any major release of Node.js.

Despite io.js’s rapid development, it has maintained or improved upon the stability of Node. io.js’s extensive test framework, frequent-but-small release cycle, and talented team of contributors have all helped to find and fix bugs faster than ever. io.js is the most stable server-side JavaScript framework today.

Lastly, we found that the effort to switch from Node v0.12 to io.js was quite small. Since we were already planning on upgrading to Node v0.12, the benefits of io.js were worth the slight increase in work. Ours was a bit of a special case, as we had to do tweak a few of our native modules. For most users, io.js is a drop-in replacement.

If you’re considering upgrading to Node v0.12, give io.js a try. You may be pleasantly surprised at the outcome.

If you missed our downtime page, you can always view it here.

Thanks goes to #io.js on Freenode for their suggestions and corrections.

Please note that this post should not be construed as disparaging or insulting Node.js or its contributors. I personally wish them the best of luck and hope they can regain momentum. While we prefer io.js right now, that could easily change. After all, io.js was unheard of only 90 days ago. May the best product win. </small>

New Feature: Follow Individual Users

2014-12-09T13:50:05-08:00

We have added a new feature! You can now follow changes from specific users in a workspace. Previously, it was only possible to follow everyone in a workspace. While useful, this can be distracting and hectic. If more than one person is editing at a time, you’re liable to get jumped back-and-forth between files. Some of our users requested a more fine-grained approach. Now we’ve added this to all of our editor plugins and our web-based editor.

What follows are instructions for how use this feature in our various plugins. Note: You can only follow people who have edit permission in the workspace. In the case of editor plugins, people who do not have edit permissions do not appear in the list of users you can follow. You can follow more than one user at a time. Also, it’s still possible to follow all changes made in the workspace.

Web Editor

To follow a specific user, click the magnet icon on their user image. To stop following that user, click it again. The magnet icon will reflect the current state. A solid looking magnet means you are following that user.

Sublime Text

To follow individual users in Sublime Text open up the command prompt and search for “Floobits - Follow User”, after selecting this option you will be presented with a list of usernames that you can follow. You should see the username of the person you are following in your status bar. To unfollow a user select “Floobits - Follow User” again and click the username you wish to unfollow. You can also stop following everyone with the “Floobits - Stop Following Workspace” command.

Follow user in Sublime Text

Select user

Confirmation

Unfollow

IntelliJ, WebStorm, PyCharm, RubyMine, Android Studio, PHPStorm

In an IntelliJ based editor you can go to tools -> Floobits -> Follow Selected Users or search for Follow Selected Users from the command prompt (⌘/Cntrl+Shift+A). A list of users with edit permissions will appear. Check the box next to each username that you wish to follow. Open this same list whenever you want to unfollow anyone. To stop following completely select Toggle Follow Mode from the command prompt. You can also use the toolbar menu Tools -> Floobits -> “Disable follow mode” to achieve the same thing.

Follow user in intellij

Select user

Emacs

In emacs type M-x and select follow-user. This will present a list of users with edit permissions you can follow. floobits-follow-mode-toggle will allow you to stop following all those you’ve elected to follow.

Follow user in emacs

Select user

Confirmation

Neovim

To follow specific individuals in NeoVim type :FlooFollowUser. To unfollow all followed users type :FlooToggleFollowMode.

Follow user in Neovim

Select user

That’s it! Enjoy!

Floobits Now Supports Neovim

2014-11-04T17:19:54-08:00

We’ve made a Neovim-compatible plugin. It deprecates our Vim plugin. We have updated all of our documentation and references to Vim to reflect this. The reason for this change is simple: Neovim offers the asynchronous architecture required to make a real-time collaborative plugin painless.

We have documented our efforts with Vim previously, implementing collaboration and a setTimeout. With Neovim, this was very simple to do, requiring no hacks at all. Goodbye, cursorhold and feedkeys!

Our Neovim plugin already offers a better experience than our Vim plugin. It has fewer bugs and performs better. Neovim is still considered alpha-quality, but we’ve found it works identically to Vim. Moving a .vimrc to .nvimrc and .vim to .nvim worked without any problems. Building Neovim on a Mac is painless.

The biggest drawback is that Pathogen is no longer supported because of a bug handling special python directories. Fortunately, Vundle works perfectly with Neovim, and makes it very easy to keep your plugins up to date.

We’ll keep our Vim plugin supported as best as we can, but we want new users to avoid it. If you’re a Vim user, try out Neovim! It’s a much better experience.

Building an Office in Software

2014-07-22T15:21:12-07:00

We use Floobits to develop Floobits, so we encounter many of the same pain points as our users. One problem we’ve had is chat. Often, we want to be able to communicate without necessarily working on the same code. Our first solution was to designate one workspace for chatting (both text and video). This worked, but the experience was definitely sub-optimal.

In an office, it’s easy to tell when someone doesn’t want to be interrupted. When they’re not “in the zone”, you can walk up and ask for help or information. If you want to have a conversation without distracting others, there are meeting rooms. Staying in the same Google+ Hangout or keeping video chat open all day is a very different experience. Unlike in an office, a conversation between two people distracts everyone. We needed a way to talk to each other without forcing our coworkers to leave the video chat or mute their sound. So we built a new feature: Org chat.

Every Floobits organization now has a chat page, which can only be accessed by members of that organization. People using org chat have a picture that updates periodically. Clicking on that picture starts a video chat with that person. You can rope in more people by clicking on more pictures. Clicking on a video stops the chat for that user. Clicking on yourself causes you to leave video chat.^[1] There’s also text chat for less attention-grabbing communication. This interaction is much closer to being in the same room. In some ways, it’s better. People aren’t distracted by others conversing nearby.

While building this feature, we discovered it had other advantages. It uses much less bandwidth than continuous video chat. This makes it usable over metered (often mobile) connections. Also, it improves battery life for laptop users, since less time is spent powering the camera, encoding video, and transmitting it. All of these benefits have helped org chat quickly become the stardard way we interact when we’re not pairing.

If you are a member of a Floobits org, you can try org chat with your teammates right now. Just go to your org list and click on a “Chat” button. The only requirements are a camera, microphone, and a WebRTC-capable browser such as Firefox or Chrome. Even Chrome on Android works.

We like to release early and often, so consider the current org chat a first draft. We plan on adding some very useful features in the near future, including:

Screen sharing.
Indicators of who is currently video chatting.
Showing who is in each workspace.

We’re also toying with adding notes and an IRC bridge, but those features are farther off.

It’s been almost a year since I wrote, “…any disadvantage of remote working is a software problem.” I still think that’s true. Remote collaboration software still has a long way to go, but we’re doing our best to move it forward.

Credit where credit is due. This workflow was inspired by Sqwiggle.

Dear Network Admins, Stop Blocking Non-HTTP/HTTPS

2014-07-17T15:05:14-07:00

One of the biggest problems we’ve had at Floobits is network filtering. Many network administrators block outbound connections that use protocols besides HTTP/HTTPS. It’s not uncommon for this to happen in schools, large companies, government offices, and hotels. This is troublesome.

We recently released new versions of our plugins that detect port blocking and work-around it. If our plugins can’t connect to floobits.com on port 3448, they try proxy.floobits.com on port 443. All of our plugins communicate over TLS, so the network traffic looks like HTTPS. Unless the connection is man-in-the-middled, it should work without users noticing. Building this took weeks of planning, development, and testing. This was time that should have been used for more productive purposes.

Network admins make various excuses for this censorship. It prevents abuse. It stops people from using protocols associated with piracy. These points are valid, but there are better ways to address them. More importantly, the costs far outweigh the benefits. Unfortunately, these costs are paid by users and developers, not network admins.

To give a concrete example: I recently vacationed in Canada with my parents and siblings. We stayed at one hotel that restricted outbound network access. This was particularly frustrating at the time, because my grandmother had passed away recently, and my mother was trying to get grandma’s headstone made. Because outbound SMTP was blocked, she couldn’t change a message on the headstone. She fortunately managed to find a different connection, but the point is made: one cannot fully anticipate the consequences of restricting outbound Internet access. Many users hardly notice the restrictions. Many are annoyed by them. And occasionally, someone’s quality of life is seriously affected by port blocking.

The Internet is more than just the web. There are thousands of protocols besides HTTP. Individually, each one may not be popular, but the majority of people use some of them. Blocking these protocols harms everyone. It frustrates users. It forces developers to build work-arounds. It stifles innovation in network protocols. Competent administrators can secure their networks and prevent abuse without resorting to such heavy-handed tactics. If you are a network administrator, I urge you to reconsider port blocking.

Error Reporting in Floobits

2014-07-02T13:50:05-07:00

Automated error reporting is an oft-overlooked way to improve software. Linters, tests, QA, and pairing can all reduce bugs, but they don’t answer the most important question: Do users encounter errors? Internal testing simply can’t imagine (let alone discover) the myriad ways in which users will break your code. Detecting and reporting errors gives feedback that can find gaps in your testing and QA. Error reporting also makes customers happier. Users love it when we contact them, apologize for an error they ran into, and mention that we’ve shipped a fix.

We make use of error reporting throughout our software. We receive reports from our editor plugins, client-side JavaScript, and server-side code. If anything breaks, we know.

The specifics of each form of error reporting are explained below, in the hope that others may learn from our set-up.

Django

We use a customized version of Django’s exception reporting. Instead of sending e-mails from Python, errors are added to an on-disk queue, which our Gurgitator service consumes. This lets our error page load sooner, since the HTTP response isn’t blocked by sending an e-mail.

Back-end services

Our back-end services use similar error reporting. All of our services are managed by runit. If a service dies with a non-zero exit code, the service’s finish script drops a message into Gurgitator’s job directory.

JavaScript

We use browser onerror handlers to detect JavaScript errors. We then log those errors by hitting our own API endpoint.

onerror can generate a lot of errors in quick succession — thousands in some cases — so we uniqify and throttle errors. Throttled occurrences are aggregated before being reported. We also filter out errors caused by extensions, content scripts and unsupported browsers.

Chrome sends additional helpful information such as a stack trace and the line and column number which helps since our JavaScript is minified to a single line.

With each report, we attempt to capture information about the state of the application while being sure to remove any sensitive information. We also use our client-side logging API for other events, such as timeouts that might occur when users attempt to link their editor plugin with their Floobits account on the website.

Editor Plugins

Each of our editor plugins makes use of the same client reporting API that the website uses. We use it to notify us when our plugins crash. Detecting plugin crashes can be challenging as there is no all-encompassing error event we can handle. We have to make use of try except in Python and try catch in Java to gain as wide as coverage as possible. Asynchronous calls will not be caught by a single try catch so there are multiple points where we need to handle potential exceptions.

Though some errors may not end up having been reported to us, it is likely something was logged either via the editor’s default logging system or to a shared floobits log file. If users are running into problems it always helps if they send us a log file as well as provide information about the editor they are using, its version and the operating system version, as well as the version of the floobits plugin.

Given that we are always making improvements to our plugins and fixing problems when we see them, it is good if everyone keeps their Floobits plugin up to date. The best way to do this is to install the plugin via the recommended method as documented in our help page for each supported editor. For example, Sublime Text users using Package Control and IntelliJ IDEA users using the Jetbrains plugin repository should receive automatic plugin updates or notifications. If the plugin was installed manually, users must remember to update regularly or at least when they’ve encountered an issue.

Floobits News

Floobits Atom: Fixing Bugs (in Atom)

Fixing Video Chat: Content Security Policy

Fixing Crashes: Avoiding NODE_ENV Erasure

A Side Note: Node’s process.env

Conclusion

Lodash v4.0.0: Not All Bugs are Created `isEqual()`

New Feature: Atom Chat

New Feature: Request Code Review

New Feature: Custom Highlight Colors

Bad Base64, a Not-so-Tricky Bug

The Key Clue

Our Service Architecture

Our Protocol

Piecing it Together

Conclusion

Why is Node.js Crashing? A Deep Dive into a Tricky Bug

Developing Atom Packages, Part 2: So Much Potential, So Many Bugs

Yay, it’s a Browser. Oh no, it’s a Browser!

Packaging

The View System

Miscellaneous

Developing Atom Plugins, Part 1: On the Bleeding Edge

The History of Atom in 30 Seconds

Not All Rainbows and Unicorns

It’s a Browser… Sort Of

A Few Missing Pieces

IntelliJ IDEA Plugin Development and Java 6

Sublime Text's Plugin API: It's Python... Sort Of

Introducing Floobits Pro

New Chat Feature: Floobot

IntelliJ IDEA User Interface Updates

On Moving to io.js

New Feature: Follow Individual Users

Web Editor

Sublime Text

IntelliJ, WebStorm, PyCharm, RubyMine, Android Studio, PHPStorm

Emacs

Neovim

Floobits Now Supports Neovim

Building an Office in Software

Dear Network Admins, Stop Blocking Non-HTTP/HTTPS

Error Reporting in Floobits

Django

Back-end services

JavaScript

Editor Plugins

Fixing Crashes: Avoiding `NODE_ENV` Erasure

A Side Note: Node’s `process.env`