Every now and then, a “big deal” data loss event shows up in the news.
The reason those events are news worthy is not because they are rare, but because they have brought down companies or impeded investigations. They may have even facilitated a fraud.
The problem is bad enough when email, text documents or spreadsheets are lost. These types of data are not changed rapidly from day to day so at least the recovery method is straight-forward.
Financial systems and other enterprise applications are a bit different though. Capturing up to the minute backups is more difficult with the best of backup strategies. Repairing mistakes in a database often means the loss of work even when a solid backup policy is followed.
As a consultant, I have never lost a customer’s data, but I have cost myself some extra work once or twice. Every single time, it is because I violated one of a handful of rules I try to consistently practice. This includes having to tell the customer “No” sometimes. There are just some things I cannot (and should not) do for my customers!
In no particular order:
1. I do not book any financial entries
I am always happy to help try things for the client in the test database, but the best practice is to let the client make production entries and click the post button on their own. The entries leave audit trails and usually cannot be deleted. It is best to not have the maintenance person booking transactions.
1a. I do not book any financial entries
No really…there are no exceptions.
2. I do not change security without written permission from the controller
Security measures do not do much good if a user can just swing by and say “Hey, I need access to ____.” There are things like separation of duties and accidental disclosures of sensitive data to consider.
3. I do not make “quick” changes to the database schema without extensive testing
I just discovered today that one of the IT folks at a client made an innocent change to a database view and it broke six reports. Every change is a major change!
And speaking of changes:
4. I do not make “quick” changes without a backup
Running an SQL Server backup is just too easy to not do a quick backup before making changes. There is always a chance of defeating the business logic of an application and throwing off the data.
5. I do not open both the test environment and the production environment at the same time
No matter what software or tool I am using, it is risky to have both environments open while testing changes. The inconvenience and extra time spent is minor compared to the risk of running untested code in production.
How about you?
I know this isn’t an exhaustive list. I would love to hear about the approaches others have taken to protect the production environment during maintenance. What policies or procedures do you follow?