{"id":8259,"date":"2017-02-02T10:13:33","date_gmt":"2017-02-02T15:13:33","guid":{"rendered":"https:\/\/spinor.info\/weblog\/?p=8259"},"modified":"2017-02-02T10:13:33","modified_gmt":"2017-02-02T15:13:33","slug":"after-a-second-notices-he-ran-it-on-db1-instead-of-db2","status":"publish","type":"post","link":"https:\/\/spinor.info\/weblog\/?p=8259","title":{"rendered":"After a second notices he ran it on db1 instead of db2&#8230;"},"content":{"rendered":"<p>&#8220;After a second notices he ran it on db1 instead of db2&#8243;&#8230; This sentence (somewhat shortened, to make a fitting title) describes the beginning of a <a href=\"https:\/\/docs.google.com\/document\/d\/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik\/pub\">colossally effed up night<\/a> at GitLab.com.<\/p>\n<p><a href=\"https:\/\/about.gitlab.com\/2017\/02\/01\/gitlab-dot-com-database-incident\/\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-8260 alignright\" src=\"https:\/\/spinor.info\/weblog\/wp-content\/uploads\/2017\/02\/gitlab-incident.png\" alt=\"\" width=\"294\" height=\"201\" \/><\/a><\/p>\n<p>In response to a spike in system load, which resulted in lag on a replication server, the operator thought that maybe restarting the replication server with a clean slate is a good idea. So he decided to wipe the replication server&#8217;s data directory.<\/p>\n<p>Unfortunately, he entered the command in the wrong window.<\/p>\n<p>I feel his pain. I did make similar mistakes before, albeit on a much smaller scale, and the memories still hurt me, years later.<\/p>\n<p>I have to commend GitLab for their exceptional openness about this incident, offering us all a valuable lesson. I note that others also responded positively, offering sympathy, assistance, and useful advice.<\/p>\n<p>I read their post-mortem with great interest. In reaction, I already implemented something that I should have done years ago: changing the background color of some of the xterm windows that I regularly open to my Linux servers, to distinguish them visually. (&#8220;Create issue to change terminal PS1 format\/colours to make it clear whether you\u2019re using production or staging&#8221;).<\/p>\n<p>Of course similar incidents and near misses also changed my habits over the years. I rarely delete anything these days without making a backup first. I always pause before hitting Enter on a command that is not (easily) reversible. I have multiple backups, and tested procedures for recovery.<\/p>\n<p>Even so&#8230; as Forrest Gump says, shit happens. And every little bit helps, especially when we can learn from the valuable lessons of others without having to go through their pain.<\/p>\n<fb:like href='https:\/\/spinor.info\/weblog\/?p=8259' send='true' layout='standard' show_faces='true' width='450' height='65' action='like' colorscheme='light' font='lucida grande'><\/fb:like>","protected":false},"excerpt":{"rendered":"<p>&#8220;After a second notices he ran it on db1 instead of db2&#8243;&#8230; This sentence (somewhat shortened, to make a fitting title) describes the beginning of a colossally effed up night at GitLab.com. In response to a spike in system load, which resulted in lag on a replication server, the operator thought that maybe restarting the <a href='https:\/\/spinor.info\/weblog\/?p=8259' class='excerpt-more'>[&#8230;]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51,6],"tags":[],"class_list":["post-8259","post","type-post","status-publish","format-standard","hentry","category-computer-security","category-internet","category-51-id","category-6-id","post-seq-1","post-parity-odd","meta-position-corners","fix"],"_links":{"self":[{"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=\/wp\/v2\/posts\/8259","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8259"}],"version-history":[{"count":2,"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=\/wp\/v2\/posts\/8259\/revisions"}],"predecessor-version":[{"id":8262,"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=\/wp\/v2\/posts\/8259\/revisions\/8262"}],"wp:attachment":[{"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8259"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8259"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/spinor.info\/weblog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8259"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}