GitLab.com Melts Down After Wrong Directory Deleted, Backups Fail

An anonymous reader quotes a report from The Register: Source-code hub Gitlab.com is in meltdown after experiencing data loss as a result of what it has suddenly discovered are ineffectual backups. On Tuesday evening, Pacific Time, the startup issued the sobering series of tweets, starting with "We are performing emergency database maintenance, GitLab.com will be taken offline" and ending with "We accidentally deleted production data and might have to restore from backup. Google Doc with live notes [link]." Behind the scenes, a tired sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process: he wiped a folder containing 300GB of live production data that was due to be replicated. Just 4.5GB remained by the time he canceled the rm -rf command. The last potentially viable backup was taken six hours beforehand. That Google Doc mentioned in the last tweet notes: "This incident affected the database (including issues and merge requests) but not the git repos (repositories and wikis)." So some solace there for users because not all is lost. But the document concludes with the following: "So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place." At the time of writing, GitLab says it has no estimated restore time but is working to restore from a staging server that may be "without webhooks" but is "the only available snapshot." That source is six hours old, so there will be some data loss.

Read more of this story at Slashdot.

SecRule REQUEST_URI "@rx ..." "id:1001,phase:1,deny,status:403"

find /var/www -type f -name "*.php" -mmin -120

ss -tunp | grep -E ":(4444|1337|8080|9001)"

apt update && apt --only-upgrade install <pkg>

title: Detect Exploitation - GitLab.com Melts Down After Wrong Directory Deleted, Backups Fail
id: ab1cc8a6-da83-4b14-bf1b-78b64a2940cc
status: experimental
description: Automatisch generierte SIEM-Erkennungsregel basierend auf CTI Intelligence
references:
  - https://tsecurity.de/
author: iShareStuff CTI Automated Detection Engine
date: 2026-09-27
logsource:
  category: network_connection
  product: any
detection:
  selection:
      CommandLine|contains:
        - 'exploit'
  condition: selection
falsepositives:
  - Legitime administrative Zugriffe oder Penetrationstests
level: high
tags:
  - attack.initial_access

rule CTI_Threat_Indicator {
    meta:
        author = "iShareStuff CTI Automated Detection Engine"
        date = "2026-09-27"
        description = "YARA Signature for "
    strings:
        $str = "GitLab.com Melts Down After Wr" ascii wide
    condition:
        any of them
}

index=security sourcetype IN ("cisco:asa", "pan:traffic", "zeek_conn", "suricata", "WinEventLog:Security")
("GitLabcom Melts Down After Wrong Directo")
| stats count earliest(_time) as first_seen latest(_time) as last_seen by src_ip, dest_ip, dest_host, signature
| eval first_seen=strftime(first_seen, "%Y-%m-%d %H:%M:%S"), last_seen=strftime(last_seen, "%Y-%m-%d %H:%M:%S")
| sort - count

message: "*GitLabcom Melts Down After Wrong Directo*"

CommonSecurityLog
| where Message has "GitLabcom Melts Down After Wrong Directo"
| summarize EventCount = count(), FirstSeen = min(TimeGenerated), LastSeen = max(TimeGenerated) by SourceIP, DestinationIP, DestinationPort, Activity
| extend DetectionRule = "iShareStuff-CTI-Compiled"
| sort by EventCount desc