Bug Hunt: Only In Production

For the past few weeks, we've been working in groups on our "greenfield" projects - our chance to create something totally new, from the ground up, with a team of four. We had a few days in class to get started, a week to work while school was closed, and then one day to tidy up before a YCombinator Demo Day-style presentation to our classmates. My team built SnackReactor (which we'll release later this week) - think of it as Yelp with the knowledge of your coworkers.

Things were going great; I knocked out a few features and the site was deployed to Microsoft Azure and working great. And then, minutes before our demo, our search for restaurants stopped returning any results. Worse, the site was working on all of our development machines, but not in production. I threw together some quick debugging code, but we couldn't fix it before our demo. We presented from our development machines.

Of course, I wasn't going to let it rest there. After I got home, I first turned to the search functions, and made sure they were sending and receiving data correctly. They were.

I turned to sqlite, which we chose because of its simplicity and cost. We don't expect that SnackReactor will ever be a high traffic website (and if it becomes one, we'll work out a migration), so sqlite is not a bad choice for a database - but Azure sure doesn't like it, so it seemed like a natural culprit. But I could find no problems. I tried to move to a hosted database (ClearDB via Azure), but Bookshelf & Knex didn't want to play nicely with it.

As I was trying to get ClearDB and Knex to talk to each other, I turned on Bookshelf's debug mode, which lets me see all the queries it's actually sending to the database. And after staring at many lines of SQL output, I saw it.

By this time, it was midnight - but the query was searching for restaurants as if it was 6AM. Apparently Azure's US West servers weren't set to local time, but six hours ahead of that. Why is a total mystery - that's not UTC, it's not Eastern time - but figuring that out is for another time.

Why was this a problem? In the name of simplicity, SnackReactor only returns restaurants that are open for business at the time of the search. So because every restaurant appeared closed, the search returned no results. That's also why it was working on my development machine, which was set to local time - and why it didn't appear earlier in the day, when the restaruants were still open.

This morning, I put together a quick fix. Becuase we're getting data from the Google Places API, in which all times are local, I simply sent the local time from the client with each request - and search worked again. From now on, I'll be more mindful of the time.

SnackReactor is coming soon - watch this space for an announcement within the next week.

Bradley Portnoy

Bradley Portnoy

I'm a recovering politico and brand marketer diving head-first into the world of software development, and always searching for ways to impact the world for the better.

View Comments