One thing I can mention is that we're closer and closer to deciding that getting our science database entirely on solid state drives is going to be unavoidable if we are to ever analyze all this data. No big deal - this isn't hurting normal operations in the least.Though largely we are under normal operating conditions, there are other behind the scenes activities going on - news to come when the time is right. I guess I need that clean break point - I'm resync'ing again now and will do so again more safely next week. I thought all was well until after coming out of today's outage when the replica hit a point of confusion in its logs. I did so rather quickly though the project has been up for a while and thus not at a safe, clean break point. Except the replica database needed to be resync'ed. Neither crashed as much as the kernels got into some sort of dead lock and couldn't be wedged - in both cases we got the people down at the colocation facility to reboot the machines for us and all was well. First oscar (our main mysql server) crashed, and then a couple hours after that so did carolyn (the replica). However last week we had some scary, unexpected server crashes. Hello again! Once again I'm emerging from a span of time where I was either out of the lab or in the lab working on non-newsworthy development, and realizing it's been way too long since I drummed up one of these reports.We had our usual Tuesday outage again today.
