Benefits of Big Problems
I started a new job on September 13th and inherited a system that was developed by a third party and no one in the company had any experience with the code base. Luckily I worked for the third party in the past and had experience with some of the frameworks in use.
Even with my background I had a lot of ground to make up in order to get a complete understanding of the system. In order to get a basic understanding of the system I worked on some minor issues and got some confidence.
I eventually felt comfortable with the changes I worked on and decided to deploy them into production. Once deployed the updates did not work (you could not even log into the system). This was a BIG PROBLEM and it was my fault.
First thing was to remove the updates and revert to the previous version. DONE. Now time for the hard part figuring out what the hell happened.
I spent days going step by step through the code and analyzing what the application was doing throughout the application. I was forced to look through 90% of the code base and establish how everything worked and how the architecture was setup. This was a very tedious, time consuming, and frustrating process. When the issue was finally discovered it was a relatively easy fix and once implement the new updates were deployed with great success.
Obviously the benefit from this incident is the newly found understanding of the system and how it works. Even though the time spent on this issue was painful it forced me to gain a deeper understanding of the system, which in the long run should help.
This is not the best case scenerio. Problems like this should not occur and overall this was a failure on my part, however, I did learn a lot and will do my best for this not to happen in the future.