Growing Pains on the OpenLab

You may have noticed that the OpenLab experienced some slowness and even some downtime on Monday and Tuesday of last week, as well as this morning. Our Team immediately went to work to track down the issue and put a fix into place. We’re happy to report that the issue has been resolved and that the OpenLab is running smoothly once again.

We apologize for any frustrations or complications that arose as a result of this slowdown. We, the OpenLab Team, pride ourselves on the reliability, power, and performance of the site. When problems pop up, we work as hard as we can to address them quickly and to learn from them so that they don’t happen again.

As always, if you have any problems or concerns about the OpenLab, please contact us at OpenLab@citytech.cuny.edu. We’re here for you.

***

In the spirit of openness, we have included below the more technical details of what happened for those who are interested.

The action centered around the new “What’s happening on the OpenLab?” widget that we just introduced. We tend to roll out new features like this one during the summer, when use and the potential for disruption are at their lowest. The widget shows the ten most recent activity items that are public and are associated with a club, course, project, or portfolio.

A late-breaking issue arising from activity with future timestamps forced us to make the database query used for the widget more complex than originally intended. Unfortunately, the new query couldn’t be stress tested with the level of traffic that the OpenLab receives during the semester. The database table against which the query was being run has over two hundred thousand entries, and it hadn’t been optimized for the kind of complicated query we were throwing at it. Faculty, students, and staff returning for the Fall semester were visiting the site more and more frequently, and the widget was being loaded many times every minute. The requests started queueing for system resources, until the site ground to a halt.

It took a few days of back and forth between members of our Community and Management Teams, our developers, and the folks who manage the servers to figure out that this particular query was the bottleneck. Once we’d tracked it down, we needed a fix. In the long run, we’ll try to optimize this query to be inherently faster. But in the short run, we can work around its complexity by ensuring that it’s run infrequently. So now, when you load the “What’s Happening on the OpenLab?” widget, you’re seeing the results of a query that’s been cached, or stored away for a certain period of time, and is reused for multiple requests.