Students and faculty returning for the FA17 semester at Cogswell College were treated to a less-than-pleasant surprise upon arriving at campus. During the last weeks of the summer semester, a massive server housing students’ work along with other course materials appeared to have vanished.
Known as “The Tahoe Drive,” this server was the primary location for students to store projects and digital artwork, both completed and in-progress. This was not this first time the drive had seemingly vanished without notice (it had been silently disconnected and moved to the new campus several weeks before the college’s final semester at its former campus reached an end – temporarily disrupting access), but this disappearance was permanent. Despite constant urging by professors for students to always back up their work on a personal drive, the vast majority of student work stored on the Tahoe drive was not saved as duplicate copies, making the corruption of the server detrimental to the aspiring artists and engineers who had been diligently working for years to compile professional portfolios.
A student who prefers to stay anonymous expressed frustrations, stating “All of the work that I had done was on the server. I was thinking ‘This is great. I could put this on a portfolio.’ Like, I did a job and this is the kind of stuff I was making at that job. All gone…” While no one directly involved in dealing with the wiped drive was willing to comment on the matter, multiple interview requests led us to Evan Peebles, Systems Administrator, who was able to offer some insight.
“We had what we call a multiple redundant failure,” Peebles explained. “Tahoe has a number of drives that are each redundant. So each one has spare disks, such that if any one disk goes down, there will be one in its place. It’s more complicated than that: it's a very robust system that's used in servers around the world ... a pretty standard system. What ended up happening is that we had multiple disks go down and because multiple disks went down, we not only lost our ability to just keep going, we actually lost the data. To be precise we lost 10% of the data. Unfortunately the 10% was spread evenly so every 2MB there was a chunk of about 200KB that was missing. That caused great loss for any larger files specifically.”
It appears that due to the massive size of the server, the school did not have the means to regularly back up the files on a secondary server. “The problem with Tahoe,” Peebles clarified, “was that the storage had not been well organized and maintained by the managers of each department [and so] there was no way of cleaning it out because it was unmanaged.”
The issue compounded itself. “Unmanaged storage like this just gets bigger and bigger,” said Peebles. “The unfortunate side effect of this is that it's not a quick or easy process to back up a large, disorganized amount of data. So while there were backups, they were infrequent because the sheer scale meant backing everything up would take a week or more so we couldn't just do it whenever students were there because we had an active file being used and that could cause its own problems. While we did have several backups, they were significantly old.” Peebles added that the Tahoe drive was roughly 20 Terabytes in size, which was an added hurdle in the backup process. Andrey Fedin, Vice President of Information Technology and Campus Services, went on to state: “We did learn our lesson there as well. We put in place already some measures to make additional levels of backup which hopefully would prevent something like this if it would ever happen again”
Unfortunately, even months after the incident, it has been difficult to get a clearer picture of the details surrounding the corruption of the Tahoe drive. While the loss of work was a heavy blow in and of itself, it was the lack of clear communication from the college that seemed to frustrate the student body most. No announcement was made to students regarding the matter in any form. The school issued no apology, no statement; not even an email directing students to access the new server put up in its place.
A general polling of the student body found that, for the most part, students only discovered this was an issue when they went to look for a file that no longer existed. Many reported that they were brought up to speed by other students who had heard a variety of unconfirmed rumors regarding the corruption of the drive. Members of the faculty received notice of the occurrence – yet there is no indication that this information was to be relayed to students.
In an effort to explain the thinking behind this decision, Fedin informed our reports that “It was a miscommunication on how we are supposed to use space on shared drives” clarifying that “the primary place for students to have their coursework is Canvas … I.T. wasn’t aware that there’s some information that students keep [on Tahoe], and that’s why we didn’t send communication to students. There were some people who were supposed to use this server, like faculty for example, that did get instructions from I.T. on what to do.”
This falls in line with a perceived pattern of miscommunications between the school’s administrators and the general student body. Jerome Solomon, Dean of the College, offered a sympathetic statement in this regard: “I actually brought up and begged for there to be more communication regarding it … the things that the students experience, the faculty experiences also … so the faculty were terribly out of the loop also … so I knew that, so I was asking for broader communication … My hope really is that the I.T. department moves towards a more standardized method of communication throughout the college.” He went on to offer general advice to students who rely on the Tahoe drive as a means of storage. “I would never keep anything there. I used to say it's like a giant USB stick for the whole college … It's going to fill up; it's a mess. I wouldn’t recommend anyone to keep files on any type of directory structure that’s like that. I’ve been a big fan of trying to encourage students to use a cloud based storage to save their work.”
The anonymous student whom we spoke with drew a similar conclusion: “I don’t keep anything on the server anymore. I keep all my files on my work computer, and right now I back them up on Google Drive.”