[{"data":1,"prerenderedAt":720},["ShallowReactive",2],{"/en-us/blog/tyranny-of-the-clock/":3,"navigation-en-us":36,"banner-en-us":465,"footer-en-us":482,"Craig Miskell":692,"next-steps-en-us":705},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":26,"_id":29,"_type":30,"title":31,"_source":32,"_file":33,"_stem":34,"_extension":35},"/en-us/blog/tyranny-of-the-clock","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"6 Lessons we learned when debugging a scaling problem on GitLab.com","Get a closer look at how we investigated errors originating from scheduled jobs, and how we stumbled upon \"the tyranny of the clock.\"","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749667913/Blog/Hero%20Images/clocks.jpg","https://about.gitlab.com/blog/tyranny-of-the-clock","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"6 Lessons we learned when debugging a scaling problem on GitLab.com\",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Craig Miskell\"}],\n        \"datePublished\": \"2019-08-27\",\n      }",{"title":9,"description":10,"authors":17,"heroImage":11,"date":19,"body":20,"category":21,"tags":22},[18],"Craig Miskell","2019-08-27","Here is a story of a scaling problem on GitLab.com: How we found it,\nwrestled with it, and ultimately resolved it. And how we discovered the\ntyranny of the clock.\n\n\n## The problem\n\n\nWe started receiving reports from customers that they were intermittently\nseeing errors on Git pulls from GitLab.com, typically from CI jobs or\nsimilar automated systems. The reported error message was usually:\n\n```\n\nssh_exchange_identification: connection closed by remote host\n\nfatal: Could not read from remote repository\n\n```\n\nTo make things more difficult, the error message was intermittent and\napparently unpredictable. We weren't able to reproduce it on demand, nor\nidentify any clear indication of what was happening in graphs or logs. The\nerror message wasn't particularly helpful either; the SSH client was being\ntold the connection had gone away, but that could be due to anything: a\nflaky client or VM, a firewall we don't control, an ISP doing something\nstrange, or an application problem at our end. We deal with a *lot* of\nconnections to Git-over-SSH, in the order of ~26 million a day, or 300/s\naverage, so trying to pick out a small number of failing ones out of that\nfirehose of data was going to be difficult. It's a good thing we like a\nchallenge.\n\n\n## The first clue\n\n\nWe got in touch with one of our customers (thanks Hubert Hölzl from\nAtalanda) who was seeing the problem several times a day, which gave us a\nfoothold. Hubert was able to supply the relevant public IP address, which\nmeant we could run some packet captures on our frontend HAproxy nodes, to\nattempt to isolate the problem from a smaller data set than 'All of the SSH\ntraffic.' Even better, they were using the [alternate-ssh\nport](/blog/gitlab-dot-com-now-supports-an-alternate-git-plus-ssh-port/)\nwhich means we only had two HAProxy servers to look at, not 16.\n\n\nTrawling through these packet traces was still not fun; despite the\nconstraints, there was ~500MB of packet capture from about 6.5 hours. We\nfound the short-running connections, in which the TCP connection was\nestablished, the client sent a version string identifier, and then our\nHAProxy immediately tore down the connection with a proper TCP FIN sequence.\nThis was the first great clue. It told us that it was definitely the\nGitLab.com end that was closing the connection, not something in between the\nclient and us, meaning this was a problem we could debug.\n\n\n### Lesson #1: In Wireshark, the Statistics menu has a wealth of useful\ntools that I'd never really noticed until this endeavor.\n\n\nIn particular, 'Conversations' shows you a basic breakdown of time, packets,\nand bytes for each TCP connection in the capture, which you can sort. I\n*should* have used this at the start, instead of trawling through the\ncaptures manually. In hindsight, connections with small packet counts was\nwhat I was looking for, and the Conversations view shows this easily. I was\nthen able to use this feature to find other instances, and verify that the\nfirst instance I found was not just an unusual outlier.\n\n\n## Diving into logs\n\n\nSo what was causing HAProxy to tear down the connection to the client? It\ncertainly seemed unlikely that it was doing it arbitrarily, and there must\nbe a deeper reason; another layer of\n[turtles](https://en.wikipedia.org/wiki/Turtles_all_the_way_down), if you\nwill. The HAProxy logs seemed like the next place to check. Ours are\nstored/available in GCP BigQuery, which is handy because there's a lot of\nthem, and we needed to slice 'n dice them in lots of different ways. But\nfirst, we were able to identify the log entry for one of the incidents from\nthe packet capture, based on time and TCP ports, which was a major\nbreakthrough. The most interesting detail in that entry was the `t_state`\n(Termination State) attribute, which was `SD`. From the HAProxy\ndocumentation:\n\n```\n    S: aborted by the server, or the server explicitly refused it\n    D: the session was in the DATA phase.\n```\n\n`D` is pretty clear; the TCP connection had been properly established, and\ndata was being sent, which matched the packet capture evidence. The `S`\nmeans HAProxy received an RST, or an ICMP failure message from the backend.\nThere was no immediate clue as to which case was occurring or possible\ncauses. It could be anything from a networking issue (e.g. glitch or\ncongestion) to an application-level problem. Using BigQuery to aggregate by\nthe Git backends, it was clear it wasn't specific to any VM. We needed more\ninformation.\n\n\nSide note: It turned out that logs with `SD` weren't unique to the problem\nwe were seeing. On the alternate-ssh port we get a lot of scanning for\nHTTPS, which leads to `SD` being logged when the SSH server sees a TLS\nClientHello message while expecting an SSH greeting. This created a brief\ndetour in our investigation.\n\n\nOn capturing some traffic between HAProxy and the Git server and using the\nWireshark statistics tools again, it was quickly obvious that SSHD on the\nGit server was tearing down the connection with a TCP FIN-ACK immediately\nafter the TCP three-way handshake; HAProxy still hadn't sent the first data\npacket but was about to, and when it did very shortly after, the Git server\nresponded with a TCP RST. And thus we had the reason for HAProxy to log a\nconnection failure with `SD`. SSH was closing the connection, apparently\ndeliberately and cleanly, with the RST being just an artifact of the SSH\nserver receiving a packet after the FIN-ACK, and doesn't mean anything else\nhere.\n\n\n## An illuminating graph\n\n\nWhile watching and analyzing the `SD` logs in BigQuery, it became apparent\nthat there was quite a bit of clustering going on in the time dimension,\nwith spikes in the first 10 seconds after the top of each minute, peaking at\nabout 5-6 seconds past:\n\n\n![Connection errors grouped by\nsecond](https://gitlab.com/gitlab-com/gl-infra/infrastructure/uploads/72cd1b763c51781fa4224495f059afb5/image.png){:\n.shadow.medium.center}\n\nConnection errors, grouped by second-of-the-minute\n\n{: .note.text-center}\n\n\nThis graph is created from data collated over a number of hours, so the fact\nthat the pattern is so substantial suggests the cause is consistent across\nminutes and hours, and possibly even worse at specific times of the day.\nEven more interesting, the average spike is 3x the base load, which means we\nhave a fun scaling problem and simply provisioning 'more resource' in terms\nof VMs to meet the peak loads would potentially be prohibitively expensive.\nThis also suggested that we were hitting some hard limit, and was our first\nclue to an underlying systemic problem, which I have called \"the tyranny of\nthe clock.\"\n\n\nCron, or similar scheduling systems, often don't have sub-minute accuracy,\nand if they do, it isn't used very often because humans prefer to think\nabout things in round numbers. Consequently, jobs will run at the start of\nthe minute or hour or at other nice round numbers. If they take a couple of\nseconds to do any preparations before they do a `git fetch` from GitLab.com,\nthis would explain the connection pattern with increases a few seconds into\nthe minute, and thus the increase in errors around those times.\n\n\n### Lesson #2: Apparently a lot of people have time synchronization (via NTP\nor otherwise) set up properly.\n\n\nIf they hadn't, this problem wouldn't have emerged so clearly. Yay for NTP!\n\n\nSo what could be causing SSH to drop the connection?\n\n\n## Getting close\n\n\nLooking through the documentation for SSHD, we found MaxStartups, which\ncontrols the maximum number of connections that can be in the\npre-authenticated state. At the top of the minute, under the stampeding herd\nof scheduled jobs from around the internet, it seems plausible that we were\nexceeding the connections limit. MaxStartups actually has three numbers: the\nlow watermark (the number at which it starts dropping connections), a\npercentage of connections to (randomly) drop for any connections above the\nlow watermark, and an absolute maximum above which all new connections are\ndropped. The default is 10:30:100, and our setting at this time was\n100:30:200, so clearly we had increased the connections in the past. Perhaps\nit was time to increase it again.\n\n\nSomewhat annoyingly, the version of openssh on our servers is 7.2, and the\nonly way to see that MaxStartups is being breached in that version is to\nturn on Debug level logging. This is an absolute firehose of data, so we\ncarefully turned it on for a short period on only one server. Thankfully\nwithin a couple of minutes it was obvious that MaxStartups was being\nbreached, and connections were being dropped early as a result,.\n\n\nIt turns out that OpenSSH 7.6 (the version that comes with Ubuntu 18.04) has\nbetter logging about MaxStartups; it only requires Verbose logging to get\nit. While not ideal, it's better than Debug level.\n\n\n### Lesson #3: It is polite to log interesting information at default levels\nand deliberately dropping a connection for any reason is definitely\ninteresting to system administrators.\n\n\nSo now that we have a cause for the problem, how can we address it? We can\nbump MaxStartups, but what will that cost? Definitely a small bit of memory,\nbut would it cause any untoward downstream effects? We could only speculate,\nso we had to just try it. We bumped the value to 150:30:300 (a 50%\nincrease). This had a great positive effect, and no visible negative effect\n(such as increased CPU load):\n\n\n![Before and after\ngraph](https://gitlab.com/gitlab-com/gl-infra/production/uploads/047a4859caafc6681c9d034c202418b9/image.png){:\n.shadow.medium.center}\n\n\nBefore and after bumping MaxStartups by 50%\n\n{: .note.text-center}\n\n\nNote the substantial reduction after 01:15. We've clearly eliminated a large\nproportion of the errors, although a non-trivial amount remained.\nInterestingly, these are clustered around round numbers: the top of the\nhour, every 30 minutes, 15 minutes, and 10 minutes. Clearly the tyranny of\nthe clock continues. The top of the hour saw the biggest peaks, which seems\nreasonable in hindsight; a lot of people will simply schedule their jobs to\nrun every hour at 0 minutes past the hour. This finding was more evidence\nthat confirms our theory that it was scheduled jobs causing the spikes, and\nthat we were on the right path with this error being due to a numerical\nlimit.\n\n\nDelightfully, there were no obvious negative effects. CPU usage on the SSH\nservers stayed about the same and didn't cause any noticeable increase in\nload. Even though we were unleashing more connections that would previously\nhave been dropped, and doing so at the busiest times. This was promising.\n\n\n## Rate limiting\n\n\nAt this point we weren't keen on simply bumping MaxStartups higher; while\nour 50% increase to-date had worked, it felt pretty crude to keep on pushing\nthis arbitrarily higher. Surely there was something else we could do.\n\n\nMy search took me to the HAProxy layer that we have in front of the SSH\nservers. HAProxy has a nice 'rate-limit sessions' option for its frontend\nlisteners. When configured, it constrains the new TCP connections per-second\nthat the frontend will pass through to backends, and leaves additional\nincoming connections on the TCP socket. If the incoming rate exceeds the\nlimit (measured every millisecond) the new connections are simply delayed.\nThe TCP client (SSH in this case) simply sees a delay before the TCP\nconnection is established, which is delightfully graceful, in my opinion. As\nlong as the overall rate never spiked too high above the limit for too long,\nwe'd be fine.\n\n\nThe next question was what number we should use. This is complicated by the\nfact that we have 27 SSH backends, and 18 HAproxy frontends (16 main, two\nalt-ssh), and the frontends don't coordinate amongst themselves for this\nrate limiting. We also had to take into account how long it takes a new SSH\nsession to make it past authentication: Assuming MaxStartups of 150, if the\nauth phase took two seconds we could only send 75 new sessions per second to\nthe each backend. The [note on the\nissue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7168#note_191678023)\nhas the derivation of the math, and I won't recount it in detail here,\nexcept to note that there are four quantities needed to calculate the\nrate-limit: the counts of both server types, the value of MaxStartups, and\n`T`, which is how long the SSH session takes to auth. `T` is critical, but\nwe could only estimate it. You might speculate how well I did at this\nestimate, but that would spoil the story. I went with two seconds for now,\nand came to a rate limit per frontend of approximately 112.5, and rounded\ndown to 110.\n\n\nWe deployed. Everything was happy, yes? Errors tended to zero, and children\ndanced happily in the streets? Well, not so much. This change had no visible\neffect on the error rates. I will be honest here, and say I was rather\ndistressed. We had missed something important, or misunderstood the problem\nspace entirely.\n\n\nSo we went back to logs (and eventually the HAProxy metrics), and were able\nto verify that the rate limiting was at least working to limit to the number\nwe specified, and that historically this number had been higher, so we were\nsuccessfully constraining the rate at which connections were being\ndispatched. But clearly the rate was still too high, and not only that, it\nwasn't even *close* enough to the right number to have a measurable impact.\nLooking at the selection of backends (as logged by HAproxy) showed an\noddity: At the top of the hour, the backend connections were not evenly\ndistributed across all the SSH servers. In the sample time chosen, it varied\nfrom 30 to 121 in a given second, meaning our load balancing wasn't very\nbalanced. Reviewing the configuration showed we were using `balance source`,\nso that a given client IP address would always connect to the same backend.\nThis might be good if you needed session stickiness, but this is SSH and we\nhave no such need. It was deliberately chosen some time ago, but there was\nno record as to why. We couldn't come up with a good reason to keep it, so\nwe tried changing to leastconn, which distributes new incoming connections\nto the backend with the least number of current connections. This was the\nresult, of the CPU usage on our SSH (Git) fleet:\n\n\n![Leastconn before and\nafter](https://gitlab.com/gitlab-com/gl-infra/infrastructure/uploads/b006877c1e45ad0255a316a96750402c/before-after-leastconn-change.png){:\n.shadow.medium.center}\n\n\nBefore and after turning on leastconn\n\n{: .note.text-center}\n\n\nClearly leastconn was a good idea. The two low-usage lines are our\n[Canary](/handbook/engineering/infrastructure/library/canary/) servers and\ncan be ignored, but the spread on the others before the change was 2:1 (30%\nto 60%), so clearly some of our backends were much busier than others due to\nthe source IP hashing. This was surprising to me; it seemed reasonable to\nexpect the range of client IPs to be sufficient to spread the load much more\nevenly, but apparently a few large outliers were enough to skew the usage\nsignificantly.\n\n\n### Lesson #4: When you choose specific non-default settings, leave a\ncomment or link to documentation/issues as to why, future people will thank\nyou.\n\n This transparency is [one of GitLab's core values](https://handbook.gitlab.com/handbook/values/#say-why-not-just-what).\n\nTurning on leastconn also helped reduce the error rates, so it is something\nwe wanted to continue with. In the spirit of experimenting, we dropped the\nrate limit lower to 100, which further reduced the error rate, suggesting\nthat perhaps the initial estimate for `T` was wrong. But if so, it was too\nsmall, leading to the rate limit being too high, and even 100/s felt pretty\nlow and we weren't keen to drop it further. Unfortunately for some\noperational reasons these two changes were just an experiment, and we had to\nroll back to `balance source` and rate limit of 100.\n\n\nWith the rate limit as low as we were comfortable with, and leastconn\ninsufficient, we tried increasing MaxStartups: first to 200 with some\neffect, then to 250. Lo, the errors all but disappeared, and nothing bad\nhappened.\n\n\n### Lesson #5: As scary as it looks, MaxStartups appears to have very little\nperformance impact even if it's raised much higher than the default.\n\n\nThis is probably a large and powerful lever we can pull in future, if\nnecessary. It's possible we might notice problems if it gets into the\nthousands or tens of thousands, but we're a long way from that.\n\n\nWhat does this say about my estimate for `T`, the time to establish and\nauthenticate an SSH session? Reverse engineering the equation, knowing that\n200 wasn't quite enough for MaxStartups, and 250 is enough, we could\ncalculate that `T` is probably between 2.7 and 3.4 seconds. So the estimate\nof two seconds wasn't far off, but the actual value was definitely higher\nthan expected. We'll come back to this a bit later.\n\n\n## Final steps\n\n\nLooking at the logs again in hindsight, and after some contemplation, we\ndiscovered that we could identify this specific failure with t_state being\n`SD` and b_read (bytes read by client) of 0. As noted above, we handle\napproximately 26-28 million SSH connections per day. It was unpleasant to\ndiscover that at the worst of the problem, roughly 1.5% of those connections\nwere being dropped badly. Clearly the problem was bigger than we had\nrealised at the start. There was nothing about this that we couldn't have\nidentified earlier (right back when we discovered that t_state=\"SD\" was\nindicative of the issue), but we didn't think to do so, and we should have.\nIt might have increased how much effort we put in.\n\n\n### Lesson #6: Measure the actual rate of your errors as early as possible.\n\n\nWe might have put a higher priority on this earlier had we realized the\nextent of the problem, although it was still dependent on knowing the\nidentifying characteristic.\n\n\nOn the plus side, after our bumps to MaxStartups and rate limiting, the\nerror rate was down to 0.001%, or a few thousand per day. This was better,\nbut still higher than we liked. After we unblocked some other operational\nmatters, we were able to formally deploy the leastconn change, and the\nerrors were eliminated entirely. We could breathe easy again.\n\n\n## Further work\n\n\nClearly the SSH authentication phase is still taking quite a while, perhaps\nup to 3.4 seconds. GitLab can use\n[AuthorizedKeysCommand](https://docs.gitlab.com/ee/administration/operations/fast_ssh_key_lookup.html)\nto look up the SSH key directly in the database. This is critical for speedy\noperations when you have a large number of users, otherwise SSHD has to\nsequentially read a very large `authorized_keys` file to look up the public\nkey of the user, and this doesn't scale well. We implement the lookup with a\nlittle bit of ruby that calls an internal HTTP API. [Stan\nHu](/company/team/#stanhu), engineering fellow and our resident source of\nGitLab knowledge, identified that the unicorn instances on the Git/SSH\nservers were experiencing substantial queuing. This could be a significant\ncontributor to the ~3-second pre-authentication stage, and therefore\nsomething we need to look at further, so investigations continue. We may\nincrease the number of unicorn (or puma) workers on these nodes, so there's\nalways a worker available for SSH. However, that isn't without risk, so we\nwill need to be careful and measure well. Work continues, but slower now\nthat the core user problem has been mitigated. We may eventually be able to\nreduce MaxStartups, although given the lack of negative impact it seems to\nhave, there's little need. It would make everyone more comfortable if\nOpenSSH let us see the how close we were to hitting MaxStartups at any\npoint, rather than having to go in blind and only find out we were close\nwhen the limit is breached and connections are dropped.\n\n\nWe also need to alert when we see HAProxy logs that indicate the problem is\noccurring, because in practice there's no reason it should ever happen. If\nit does, we need to increase MaxStartups further, or if resources are\nconstrained, add more Git/SSH nodes.\n\n\n## Conclusion\n\n\nComplex systems have complex interactions, and there is often more than one\nlever that can be used to control various bottlenecks. It's good to know\nwhat tools are available because they often have trade-offs. Assumptions and\nestimates can also be risky. In hindsight, I would have attempted to get a\nmuch better measurement of how long authentication takes, so that my `T`\nestimate was better.\n\n\nBut the biggest lesson is that when large numbers of people schedule jobs at\nround numbers on the clock, it leads to really interesting scaling problems\nfor centralized service providers like GitLab. If you're one of them, you\nmight like to consider putting in a random sleep of maybe 30 seconds at the\nstart, or pick a random time during the hour *and* put in the random sleep,\njust to be polite and fight the tyranny of the clock.\n\n\nCover image by [Jon Tyson](https://unsplash.com/@jontyson) on\n[Unsplash](https://unsplash.com)\n\n{: .note}\n","engineering",[23,24,25],"git","performance","production",{"slug":27,"featured":6,"template":28},"tyranny-of-the-clock","BlogPost","content:en-us:blog:tyranny-of-the-clock.yml","yaml","Tyranny Of The Clock","content","en-us/blog/tyranny-of-the-clock.yml","en-us/blog/tyranny-of-the-clock","yml",{"_path":37,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":39,"_id":461,"_type":30,"title":462,"_source":32,"_file":463,"_stem":464,"_extension":35},"/shared/en-us/main-navigation","en-us",{"logo":40,"freeTrial":45,"sales":50,"login":55,"items":60,"search":392,"minimal":423,"duo":442,"pricingDeployment":451},{"config":41},{"href":42,"dataGaName":43,"dataGaLocation":44},"/","gitlab logo","header",{"text":46,"config":47},"Get free trial",{"href":48,"dataGaName":49,"dataGaLocation":44},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":51,"config":52},"Talk to sales",{"href":53,"dataGaName":54,"dataGaLocation":44},"/sales/","sales",{"text":56,"config":57},"Sign in",{"href":58,"dataGaName":59,"dataGaLocation":44},"https://gitlab.com/users/sign_in/","sign in",[61,105,203,208,313,373],{"text":62,"config":63,"cards":65,"footer":88},"Platform",{"dataNavLevelOne":64},"platform",[66,72,80],{"title":62,"description":67,"link":68},"The most comprehensive AI-powered DevSecOps Platform",{"text":69,"config":70},"Explore our Platform",{"href":71,"dataGaName":64,"dataGaLocation":44},"/platform/",{"title":73,"description":74,"link":75},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":76,"config":77},"Meet GitLab Duo",{"href":78,"dataGaName":79,"dataGaLocation":44},"/gitlab-duo/","gitlab duo ai",{"title":81,"description":82,"link":83},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":84,"config":85},"Learn more",{"href":86,"dataGaName":87,"dataGaLocation":44},"/why-gitlab/","why gitlab",{"title":89,"items":90},"Get started with",[91,96,101],{"text":92,"config":93},"Platform Engineering",{"href":94,"dataGaName":95,"dataGaLocation":44},"/solutions/platform-engineering/","platform engineering",{"text":97,"config":98},"Developer Experience",{"href":99,"dataGaName":100,"dataGaLocation":44},"/developer-experience/","Developer experience",{"text":102,"config":103},"MLOps",{"href":104,"dataGaName":102,"dataGaLocation":44},"/topics/devops/the-role-of-ai-in-devops/",{"text":106,"left":107,"config":108,"link":110,"lists":114,"footer":185},"Product",true,{"dataNavLevelOne":109},"solutions",{"text":111,"config":112},"View all Solutions",{"href":113,"dataGaName":109,"dataGaLocation":44},"/solutions/",[115,140,164],{"title":116,"description":117,"link":118,"items":123},"Automation","CI/CD and automation to accelerate deployment",{"config":119},{"icon":120,"href":121,"dataGaName":122,"dataGaLocation":44},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[124,128,132,136],{"text":125,"config":126},"CI/CD",{"href":127,"dataGaLocation":44,"dataGaName":125},"/solutions/continuous-integration/",{"text":129,"config":130},"AI-Assisted Development",{"href":78,"dataGaLocation":44,"dataGaName":131},"AI assisted development",{"text":133,"config":134},"Source Code Management",{"href":135,"dataGaLocation":44,"dataGaName":133},"/solutions/source-code-management/",{"text":137,"config":138},"Automated Software Delivery",{"href":121,"dataGaLocation":44,"dataGaName":139},"Automated software delivery",{"title":141,"description":142,"link":143,"items":148},"Security","Deliver code faster without compromising security",{"config":144},{"href":145,"dataGaName":146,"dataGaLocation":44,"icon":147},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[149,154,159],{"text":150,"config":151},"Application Security Testing",{"href":152,"dataGaName":153,"dataGaLocation":44},"/solutions/application-security-testing/","Application security testing",{"text":155,"config":156},"Software Supply Chain Security",{"href":157,"dataGaLocation":44,"dataGaName":158},"/solutions/supply-chain/","Software supply chain security",{"text":160,"config":161},"Software Compliance",{"href":162,"dataGaName":163,"dataGaLocation":44},"/solutions/software-compliance/","software compliance",{"title":165,"link":166,"items":171},"Measurement",{"config":167},{"icon":168,"href":169,"dataGaName":170,"dataGaLocation":44},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[172,176,180],{"text":173,"config":174},"Visibility & Measurement",{"href":169,"dataGaLocation":44,"dataGaName":175},"Visibility and Measurement",{"text":177,"config":178},"Value Stream Management",{"href":179,"dataGaLocation":44,"dataGaName":177},"/solutions/value-stream-management/",{"text":181,"config":182},"Analytics & Insights",{"href":183,"dataGaLocation":44,"dataGaName":184},"/solutions/analytics-and-insights/","Analytics and insights",{"title":186,"items":187},"GitLab for",[188,193,198],{"text":189,"config":190},"Enterprise",{"href":191,"dataGaLocation":44,"dataGaName":192},"/enterprise/","enterprise",{"text":194,"config":195},"Small Business",{"href":196,"dataGaLocation":44,"dataGaName":197},"/small-business/","small business",{"text":199,"config":200},"Public Sector",{"href":201,"dataGaLocation":44,"dataGaName":202},"/solutions/public-sector/","public sector",{"text":204,"config":205},"Pricing",{"href":206,"dataGaName":207,"dataGaLocation":44,"dataNavLevelOne":207},"/pricing/","pricing",{"text":209,"config":210,"link":212,"lists":216,"feature":300},"Resources",{"dataNavLevelOne":211},"resources",{"text":213,"config":214},"View all resources",{"href":215,"dataGaName":211,"dataGaLocation":44},"/resources/",[217,250,272],{"title":218,"items":219},"Getting started",[220,225,230,235,240,245],{"text":221,"config":222},"Install",{"href":223,"dataGaName":224,"dataGaLocation":44},"/install/","install",{"text":226,"config":227},"Quick start guides",{"href":228,"dataGaName":229,"dataGaLocation":44},"/get-started/","quick setup checklists",{"text":231,"config":232},"Learn",{"href":233,"dataGaLocation":44,"dataGaName":234},"https://university.gitlab.com/","learn",{"text":236,"config":237},"Product documentation",{"href":238,"dataGaName":239,"dataGaLocation":44},"https://docs.gitlab.com/","product documentation",{"text":241,"config":242},"Best practice videos",{"href":243,"dataGaName":244,"dataGaLocation":44},"/getting-started-videos/","best practice videos",{"text":246,"config":247},"Integrations",{"href":248,"dataGaName":249,"dataGaLocation":44},"/integrations/","integrations",{"title":251,"items":252},"Discover",[253,258,262,267],{"text":254,"config":255},"Customer success stories",{"href":256,"dataGaName":257,"dataGaLocation":44},"/customers/","customer success stories",{"text":259,"config":260},"Blog",{"href":261,"dataGaName":5,"dataGaLocation":44},"/blog/",{"text":263,"config":264},"Remote",{"href":265,"dataGaName":266,"dataGaLocation":44},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":268,"config":269},"TeamOps",{"href":270,"dataGaName":271,"dataGaLocation":44},"/teamops/","teamops",{"title":273,"items":274},"Connect",[275,280,285,290,295],{"text":276,"config":277},"GitLab Services",{"href":278,"dataGaName":279,"dataGaLocation":44},"/services/","services",{"text":281,"config":282},"Community",{"href":283,"dataGaName":284,"dataGaLocation":44},"/community/","community",{"text":286,"config":287},"Forum",{"href":288,"dataGaName":289,"dataGaLocation":44},"https://forum.gitlab.com/","forum",{"text":291,"config":292},"Events",{"href":293,"dataGaName":294,"dataGaLocation":44},"/events/","events",{"text":296,"config":297},"Partners",{"href":298,"dataGaName":299,"dataGaLocation":44},"/partners/","partners",{"backgroundColor":301,"textColor":302,"text":303,"image":304,"link":308},"#2f2a6b","#fff","Insights for the future of software development",{"altText":305,"config":306},"the source promo card",{"src":307},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":309,"config":310},"Read the latest",{"href":311,"dataGaName":312,"dataGaLocation":44},"/the-source/","the source",{"text":314,"config":315,"lists":317},"Company",{"dataNavLevelOne":316},"company",[318],{"items":319},[320,325,331,333,338,343,348,353,358,363,368],{"text":321,"config":322},"About",{"href":323,"dataGaName":324,"dataGaLocation":44},"/company/","about",{"text":326,"config":327,"footerGa":330},"Jobs",{"href":328,"dataGaName":329,"dataGaLocation":44},"/jobs/","jobs",{"dataGaName":329},{"text":291,"config":332},{"href":293,"dataGaName":294,"dataGaLocation":44},{"text":334,"config":335},"Leadership",{"href":336,"dataGaName":337,"dataGaLocation":44},"/company/team/e-group/","leadership",{"text":339,"config":340},"Team",{"href":341,"dataGaName":342,"dataGaLocation":44},"/company/team/","team",{"text":344,"config":345},"Handbook",{"href":346,"dataGaName":347,"dataGaLocation":44},"https://handbook.gitlab.com/","handbook",{"text":349,"config":350},"Investor relations",{"href":351,"dataGaName":352,"dataGaLocation":44},"https://ir.gitlab.com/","investor relations",{"text":354,"config":355},"Trust Center",{"href":356,"dataGaName":357,"dataGaLocation":44},"/security/","trust center",{"text":359,"config":360},"AI Transparency Center",{"href":361,"dataGaName":362,"dataGaLocation":44},"/ai-transparency-center/","ai transparency center",{"text":364,"config":365},"Newsletter",{"href":366,"dataGaName":367,"dataGaLocation":44},"/company/contact/","newsletter",{"text":369,"config":370},"Press",{"href":371,"dataGaName":372,"dataGaLocation":44},"/press/","press",{"text":374,"config":375,"lists":376},"Contact us",{"dataNavLevelOne":316},[377],{"items":378},[379,382,387],{"text":51,"config":380},{"href":53,"dataGaName":381,"dataGaLocation":44},"talk to sales",{"text":383,"config":384},"Get help",{"href":385,"dataGaName":386,"dataGaLocation":44},"/support/","get help",{"text":388,"config":389},"Customer portal",{"href":390,"dataGaName":391,"dataGaLocation":44},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":393,"login":394,"suggestions":401},"Close",{"text":395,"link":396},"To search repositories and projects, login to",{"text":397,"config":398},"gitlab.com",{"href":58,"dataGaName":399,"dataGaLocation":400},"search login","search",{"text":402,"default":403},"Suggestions",[404,406,410,412,416,420],{"text":73,"config":405},{"href":78,"dataGaName":73,"dataGaLocation":400},{"text":407,"config":408},"Code Suggestions (AI)",{"href":409,"dataGaName":407,"dataGaLocation":400},"/solutions/code-suggestions/",{"text":125,"config":411},{"href":127,"dataGaName":125,"dataGaLocation":400},{"text":413,"config":414},"GitLab on AWS",{"href":415,"dataGaName":413,"dataGaLocation":400},"/partners/technology-partners/aws/",{"text":417,"config":418},"GitLab on Google Cloud",{"href":419,"dataGaName":417,"dataGaLocation":400},"/partners/technology-partners/google-cloud-platform/",{"text":421,"config":422},"Why GitLab?",{"href":86,"dataGaName":421,"dataGaLocation":400},{"freeTrial":424,"mobileIcon":429,"desktopIcon":434,"secondaryButton":437},{"text":425,"config":426},"Start free trial",{"href":427,"dataGaName":49,"dataGaLocation":428},"https://gitlab.com/-/trials/new/","nav",{"altText":430,"config":431},"Gitlab Icon",{"src":432,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":430,"config":435},{"src":436,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":438,"config":439},"Get Started",{"href":440,"dataGaName":441,"dataGaLocation":428},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":443,"mobileIcon":447,"desktopIcon":449},{"text":444,"config":445},"Learn more about GitLab Duo",{"href":78,"dataGaName":446,"dataGaLocation":428},"gitlab duo",{"altText":430,"config":448},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":450},{"src":436,"dataGaName":433,"dataGaLocation":428},{"freeTrial":452,"mobileIcon":457,"desktopIcon":459},{"text":453,"config":454},"Back to pricing",{"href":206,"dataGaName":455,"dataGaLocation":428,"icon":456},"back to pricing","GoBack",{"altText":430,"config":458},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":460},{"src":436,"dataGaName":433,"dataGaLocation":428},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":466,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"title":467,"button":468,"image":473,"config":477,"_id":479,"_type":30,"_source":32,"_file":480,"_stem":481,"_extension":35},"/shared/en-us/banner","is now in public beta!",{"text":469,"config":470},"Try the Beta",{"href":471,"dataGaName":472,"dataGaLocation":44},"/gitlab-duo/agent-platform/","duo banner",{"altText":474,"config":475},"GitLab Duo Agent Platform",{"src":476},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":478},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":483,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":484,"_id":688,"_type":30,"title":689,"_source":32,"_file":690,"_stem":691,"_extension":35},"/shared/en-us/main-footer",{"text":485,"source":486,"edit":492,"contribute":497,"config":502,"items":507,"minimal":680},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":487,"config":488},"View page source",{"href":489,"dataGaName":490,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":493,"config":494},"Edit this page",{"href":495,"dataGaName":496,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":498,"config":499},"Please contribute",{"href":500,"dataGaName":501,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":503,"facebook":504,"youtube":505,"linkedin":506},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[508,531,587,616,650],{"title":62,"links":509,"subMenu":514},[510],{"text":511,"config":512},"DevSecOps platform",{"href":71,"dataGaName":513,"dataGaLocation":491},"devsecops platform",[515],{"title":204,"links":516},[517,521,526],{"text":518,"config":519},"View plans",{"href":206,"dataGaName":520,"dataGaLocation":491},"view plans",{"text":522,"config":523},"Why Premium?",{"href":524,"dataGaName":525,"dataGaLocation":491},"/pricing/premium/","why premium",{"text":527,"config":528},"Why Ultimate?",{"href":529,"dataGaName":530,"dataGaLocation":491},"/pricing/ultimate/","why ultimate",{"title":532,"links":533},"Solutions",[534,539,541,543,548,553,557,560,564,569,571,574,577,582],{"text":535,"config":536},"Digital transformation",{"href":537,"dataGaName":538,"dataGaLocation":491},"/topics/digital-transformation/","digital transformation",{"text":150,"config":540},{"href":152,"dataGaName":150,"dataGaLocation":491},{"text":139,"config":542},{"href":121,"dataGaName":122,"dataGaLocation":491},{"text":544,"config":545},"Agile development",{"href":546,"dataGaName":547,"dataGaLocation":491},"/solutions/agile-delivery/","agile delivery",{"text":549,"config":550},"Cloud transformation",{"href":551,"dataGaName":552,"dataGaLocation":491},"/topics/cloud-native/","cloud transformation",{"text":554,"config":555},"SCM",{"href":135,"dataGaName":556,"dataGaLocation":491},"source code management",{"text":125,"config":558},{"href":127,"dataGaName":559,"dataGaLocation":491},"continuous integration & delivery",{"text":561,"config":562},"Value stream management",{"href":179,"dataGaName":563,"dataGaLocation":491},"value stream management",{"text":565,"config":566},"GitOps",{"href":567,"dataGaName":568,"dataGaLocation":491},"/solutions/gitops/","gitops",{"text":189,"config":570},{"href":191,"dataGaName":192,"dataGaLocation":491},{"text":572,"config":573},"Small business",{"href":196,"dataGaName":197,"dataGaLocation":491},{"text":575,"config":576},"Public sector",{"href":201,"dataGaName":202,"dataGaLocation":491},{"text":578,"config":579},"Education",{"href":580,"dataGaName":581,"dataGaLocation":491},"/solutions/education/","education",{"text":583,"config":584},"Financial services",{"href":585,"dataGaName":586,"dataGaLocation":491},"/solutions/finance/","financial services",{"title":209,"links":588},[589,591,593,595,598,600,602,604,606,608,610,612,614],{"text":221,"config":590},{"href":223,"dataGaName":224,"dataGaLocation":491},{"text":226,"config":592},{"href":228,"dataGaName":229,"dataGaLocation":491},{"text":231,"config":594},{"href":233,"dataGaName":234,"dataGaLocation":491},{"text":236,"config":596},{"href":238,"dataGaName":597,"dataGaLocation":491},"docs",{"text":259,"config":599},{"href":261,"dataGaName":5,"dataGaLocation":491},{"text":254,"config":601},{"href":256,"dataGaName":257,"dataGaLocation":491},{"text":263,"config":603},{"href":265,"dataGaName":266,"dataGaLocation":491},{"text":276,"config":605},{"href":278,"dataGaName":279,"dataGaLocation":491},{"text":268,"config":607},{"href":270,"dataGaName":271,"dataGaLocation":491},{"text":281,"config":609},{"href":283,"dataGaName":284,"dataGaLocation":491},{"text":286,"config":611},{"href":288,"dataGaName":289,"dataGaLocation":491},{"text":291,"config":613},{"href":293,"dataGaName":294,"dataGaLocation":491},{"text":296,"config":615},{"href":298,"dataGaName":299,"dataGaLocation":491},{"title":314,"links":617},[618,620,622,624,626,628,630,634,639,641,643,645],{"text":321,"config":619},{"href":323,"dataGaName":316,"dataGaLocation":491},{"text":326,"config":621},{"href":328,"dataGaName":329,"dataGaLocation":491},{"text":334,"config":623},{"href":336,"dataGaName":337,"dataGaLocation":491},{"text":339,"config":625},{"href":341,"dataGaName":342,"dataGaLocation":491},{"text":344,"config":627},{"href":346,"dataGaName":347,"dataGaLocation":491},{"text":349,"config":629},{"href":351,"dataGaName":352,"dataGaLocation":491},{"text":631,"config":632},"Sustainability",{"href":633,"dataGaName":631,"dataGaLocation":491},"/sustainability/",{"text":635,"config":636},"Diversity, inclusion and belonging (DIB)",{"href":637,"dataGaName":638,"dataGaLocation":491},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":354,"config":640},{"href":356,"dataGaName":357,"dataGaLocation":491},{"text":364,"config":642},{"href":366,"dataGaName":367,"dataGaLocation":491},{"text":369,"config":644},{"href":371,"dataGaName":372,"dataGaLocation":491},{"text":646,"config":647},"Modern Slavery Transparency Statement",{"href":648,"dataGaName":649,"dataGaLocation":491},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":651,"links":652},"Contact Us",[653,656,658,660,665,670,675],{"text":654,"config":655},"Contact an expert",{"href":53,"dataGaName":54,"dataGaLocation":491},{"text":383,"config":657},{"href":385,"dataGaName":386,"dataGaLocation":491},{"text":388,"config":659},{"href":390,"dataGaName":391,"dataGaLocation":491},{"text":661,"config":662},"Status",{"href":663,"dataGaName":664,"dataGaLocation":491},"https://status.gitlab.com/","status",{"text":666,"config":667},"Terms of use",{"href":668,"dataGaName":669,"dataGaLocation":491},"/terms/","terms of use",{"text":671,"config":672},"Privacy statement",{"href":673,"dataGaName":674,"dataGaLocation":491},"/privacy/","privacy statement",{"text":676,"config":677},"Cookie preferences",{"dataGaName":678,"dataGaLocation":491,"id":679,"isOneTrustButton":107},"cookie preferences","ot-sdk-btn",{"items":681},[682,684,686],{"text":666,"config":683},{"href":668,"dataGaName":669,"dataGaLocation":491},{"text":671,"config":685},{"href":673,"dataGaName":674,"dataGaLocation":491},{"text":676,"config":687},{"dataGaName":678,"dataGaLocation":491,"id":679,"isOneTrustButton":107},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[693],{"_path":694,"_dir":695,"_draft":6,"_partial":6,"_locale":7,"content":696,"config":700,"_id":702,"_type":30,"title":18,"_source":32,"_file":703,"_stem":704,"_extension":35},"/en-us/blog/authors/craig-miskell","authors",{"name":18,"config":697},{"headshot":698,"ctfId":699},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749667372/Blog/Author%20Headshots/cmiskell-headshot.jpg","cmiskell",{"template":701},"BlogAuthor","content:en-us:blog:authors:craig-miskell.yml","en-us/blog/authors/craig-miskell.yml","en-us/blog/authors/craig-miskell",{"_path":706,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"header":707,"eyebrow":708,"blurb":709,"button":710,"secondaryButton":714,"_id":716,"_type":30,"title":717,"_source":32,"_file":718,"_stem":719,"_extension":35},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":46,"config":711},{"href":712,"dataGaName":49,"dataGaLocation":713},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":51,"config":715},{"href":53,"dataGaName":54,"dataGaLocation":713},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1758326274188]