Week in philly - Disaster recovery
Every year for work, we perform a disaster recovery validation, which allows us to validate our processes and procedures and verify we are able to bring up the critical computer resources in a disaster environment, should our main data center become unavailable. This test is done in a remote physicality in Philadelphia, so it gives us a chance to get out and travel as well.
In past years, I have been the main driver for the test for our team, so previous years this was a very stressful time for me. This year I decided to take a back seat, and let others in the team have more of an involvement in the process. I found that for both prep time and during the exercise, this made things much less stressful for me. In past years, I was very stressed for weeks before the exercise, and until a week after the exercise ended, then I would wind down. This year because I handed off many more of the components to the test, I was more of a participant than anything, and this made things much easier on me, I found I was much more relaxed this year versus previous years.
This year we got out on Tuesday June 20th, with the test starting Wednesday June 21st and running through Sunday June 25th. One thing I hate is that every year, the test seems to fall on the same weekend as the ham radio field day, which is something else I would like to participate in. Tuesday we had dinner at The Capital Grille, which was a very nice. I was very impressed with the scotch selection they had.
Wednesday morning we had to switch hotels. We had been scheduled to stay in the Residence Inn Philadelphia Center City but due to rooms being full, we spent the first night in the Sheraton Suites Philadelphia Airport. After the hotel room switch, it was off to Sungard to start the test. First for us is to get the environment handed over from the sungard tech’s, and get the backup and recovery environment put together. After that, we then do our server recoveries off of tape and disk, and then the end users do testing. All very boring stuff. This year I did not take over the environment build like I have done past years, I more went with the flow of things.
I worked the day shift for the most part, and was able to find time to go out and walk every day, which was very nice. I have found getting out and unwinding helps me quite a bit. I saw quite a bit of downtown Philly, just walking around, both during the daytime and at night. I also ate halfway decently, which makes these recovery tests bearable. In past years, food has been an issue (getting lunch on time, etc) but this year that was not a problem.
In the end, we had some issues which pushed us to the end of the window for our recovery, but that happens every year, so I am use to it at this point. It is a matter of timing things so I got time to spend time relaxing a bit between shifts I had to work, because I knew I would not have a chance to have much time to unwind at the end of the test. The week went by very quickly, and I was exhausted after things had wound up. I was asleep in my bed within a hour of getting back home from Philadelphia.