[{"data":1,"prerenderedAt":700},["ShallowReactive",2],{"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change/":3,"navigation-en-us":35,"banner-en-us":447,"footer-en-us":460,"Igor Wiedler":672,"next-steps-en-us":685},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":25,"_id":28,"_type":29,"title":30,"_source":31,"_file":32,"_stem":33,"_extension":34},"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next...","This post is about a wild discovery made while investigating strange behavior from HAProxy. We dive into the pathology, describe how we found it, and share some investigative techniques used along the way.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681844/Blog/Hero%20Images/infra-proxy-protocol-wireshark-header.png","https://about.gitlab.com/blog/this-sre-attempted-to-roll-out-an-haproxy-change","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next... \",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Igor Wiedler\"}],\n        \"datePublished\": \"2021-01-14\",\n      }",{"title":9,"description":10,"authors":17,"heroImage":11,"date":19,"body":20,"category":21,"tags":22},[18],"Igor Wiedler","2021-01-14","\n\nThis blog post was originally published on the GitLab Unfiltered blog. It was reviewed and republished on 2021-02-12.\n{: .note .alert-info .text-center}\n\n## TL;DR\n\n- HAProxy has a `server-state-file` directive that persists some of its state across restarts.\n- This state file contains the port of each backend server.\n- If an `haproxy.cfg` change modifies the port, the new port will be overwritten with the previous one from the state file.\n- A workaround is to change the backend server name, so that it is considered to be a separate server that does not match what is in the state file.\n- This has implications for the rollout procedure we use on HAProxy.\n\n## Background\n\nAll of this occurred in the context of [the gitlab-pages PROXYv2\nproject](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11902).\n\nThe rollout to staging involves changing the request flow from TCP proxying...\n```\n                   443                   443                        1443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [ web-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                        tcp            tcp,tls,http\n```\n\n... to using the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt):\n```\n                   443                   443                        2443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [ web-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                     proxyv2,tcp       proxyv2,tcp,tls,http\n```\n\nThis is done through this change to `/etc/haproxy/haproxy.cfg` on\n`fe-pages-01-lb-gstg` (note the port change):\n```diff\n-    server web-pages-01-sv-gstg web-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080\n-    server web-pages-02-sv-gstg web-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080\n+    server web-pages-01-sv-gstg web-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n+    server web-pages-02-sv-gstg web-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n```\n\nSeems straightforward enough, let's go ahead and apply that change.\n\n## The brokenness\n\nAfter applying this change on one of the two `fe-pages` nodes, the requests to\nthat node start failing.\n\nBy retrying a few times via `curl` on the command line, we see this error:\n```\n➜  ~ curl -vvv https://jarv.staging.gitlab.io/pages-test/\n*   Trying 35.229.69.78...\n* TCP_NODELAY set\n* Connected to jarv.staging.gitlab.io (35.229.69.78) port 443 (#0)\n* ALPN, offering h2\n* ALPN, offering http/1.1\n* successfully set certificate verify locations:\n*   CAfile: /etc/ssl/cert.pem\n  CApath: none\n* TLSv1.2 (OUT), TLS handshake, Client hello (1):\n* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to jarv.staging.gitlab.io:443\n* Closing connection 0\ncurl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to jarv.staging.gitlab.io:443\n```\n\nThis looks like some issue in the TLS stack, or possibly with the underlying\nconnection. It turns out that `LibreSSL` does not give us much insight into the\nunderlying issue here.\n\nSo to get a better idea, let's capture a traffic dump on the HAProxy node:\n```\nsudo tcpdump -v -w \"$(pwd)/$(hostname).$(date +%Y%m%d_%H%M%S).pcap\"\n```\n\nWhile `tcpdump` is running, we can generate some traffic, then ctrl+c and pull\nthe dump down for further analysis. That `pcap` file can be opened in Wireshark, and this allows the data to be\nexplored and filtered interactively. Here, the first really surprising thing happens:\n\n**We do not see any traffic on port 2443.**\n\nAt the same time, we _do_ see some traffic on port 1443. But we came here to look at what underlies the LibreSSL error, and what we find\nis the following (by filtering for `ip.addr == \u003Cmy external ip>`). We have a TCP SYN/ACK, establishing the connection. Followed by the client\nsending a TLS \"hello\". After which the server closes the connection with a FIN.\n\nIn other words, the server is closing the connection on the client.\n\n## The early hypotheses\n\nSo here come the usual suspects:\n\n* Did we modify the correct place in the config file?\n* Did we catch all places we need to update in the config?\n* Did the HAProxy process parse th econfig successfully?\n* Did HAProxy actually reload?\n* Is there a difference between reload and restart?\n* Did we modify the correct config file?\n* Are there old lingering HAProxy processes on the box?\n* Are we actually sending traffic to this node?\n* Are backend health checks failing?\n* Is there anything in the HAProxy logs?\n\nNone of these gave any insights whatsoever.\n\nIn an effort to reproduce the issue, I ran HAProxy on my local machine with a\nsimilar config, proxying traffic to `web-pages-01-sv-gstg`. To my surprise, this\nworked correctly. I tested with different HAProxy versions. It worked locally, but not on\n`fe-pages-01`.\n\nAt this point I'm stumped. The local config is not identical to gstg, but quite\nsimilar. What could possibly be the difference?\n\n## Digging deeper\n\nThis is when I reached out to [Matt Smiley](/company/team#/msmiley) to help with the investigation.\n\nWe started off by repeating the experiment. We saw the same results:\n\n* Server closes connection after client sends TLS hello\n* No traffic from fe-pages to web-pages on port 2443\n* Traffic from fe-pages to web-pages on port 1443\n\nThe first lead was to look at the packets going to port 1443. What do they\ncontain? We see this:\n\n![Traffic capture in wireshark showing a TCP FIN and the string QUIT in the stream](https://about.gitlab.com/images/blogimages/infra-proxy-protocol-wireshark.png){: .shadow.center}\nTraffic capture in Wireshark showing a TCP FIN and the string QUIT in the stream\n{: .note.text-center}\n\nThere is mention of `jarv.staging.gitlab.io` which does match what the client sent. And before that there is some really weird preamble:\n\n```\n\"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\"\n```\n\nWhat on earth is this? Is it from the PROXY protocol? Let's search [the\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for the word\n\"QUIT.\" Nothing.\n\nIs this something in the HAProxy source? Searching for \"QUIT\" in the code\nreveals some hits, but none that explain this.\n\nSo this is a mystery. We leave it for now, and probe in a different direction.\n\n## Honing in\n\nHow come we are sending traffic to port 1443, when that port is not mentioned in\n`haproxy.cfg`? Where on earth is HAProxy getting that information from?\n\nI suggested running `strace` on HAProxy startup, so that we can see which files\nare being `open`ed. This is a bit tricky to do though, because the process is\nsystemd-managed.\n\nIt turns out that thanks to BPF and [BCC](https://github.com/iovisor/bcc), we\ncan actually listen on open events system-wide using the wonderful\n[opensnoop](https://github.com/iovisor/bcc/blob/master/tools/opensnoop.py). So we run `opensnoop` and restart `haproxy`, and this is what we see, highlighting the relevant bit:\n```\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo /usr/share/bcc/tools/opensnoop  -T --name haproxy\n\n...\n\n24.117171000  16702  haproxy             3   0 /etc/haproxy/haproxy.cfg\n...\n24.118099000  16702  haproxy             4   0 /etc/haproxy/errors/400.http\n...\n24.118333000  16702  haproxy             4   0 /etc/haproxy/cloudflare_ips_v4.lst\n...\n24.119109000  16702  haproxy             3   0 /etc/haproxy/state/global\n```\n\nWhat do we have here? `/etc/haproxy/state/global`, this seems oddly suspicious.\nWhat could it possibly be? Let's see what this file contains.\n```\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo cat /etc/haproxy/state/global\n\n1\n# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord\n5 pages_http 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0 0 web-pages-01-sv-gstg.c.gitlab-staging-1.internal 1080 -\n5 pages_http 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0 0 web-pages-02-sv-gstg.c.gitlab-staging-1.internal 1080 -\n6 pages_https 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0 0 web-pages-01-sv-gstg.c.gitlab-staging-1.internal 1443 -\n6 pages_https 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0 0 web-pages-02-sv-gstg.c.gitlab-staging-1.internal 1443 -\n```\n\nIt appears we are storing some metadata for each backend server, including its old port number!\n\nNow, looking again in `haproxy.cfg`, we see:\n```\nglobal\n    ...\n    server-state-file /etc/haproxy/state/global\n```\n\nSo we are using the\n[`server-state-file`](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#server-state-file)\ndirective. This will persist server state across HAProxy restarts. That is\nuseful to keep metadata consistent, such as whether a server was marked as\nMAINT.\n\n**However, it appears to be clobbering the port from `haproxy.cfg`!**\n\nThe suspected behavior is:\n\n* HAProxy is running with the old config: `web-pages-01-sv-gstg`, `1443`\n* `haproxy.cfg` is updated with the new config: `web-pages-01-sv-gstg`, `2443`, `send-proxy-v2`\n* HAProxy reload is initiated\n* HAProxy writes out the state to `/etc/haproxy/state/global` (including the old port of each backend server)\n* HAProxy starts up, reads `haproxy.cfg`, initializes itself with the new config: `web-pages-01-sv-gstg`, `2443`, `send-proxy-v2`\n* HAProxy reads the state from `/etc/haproxy/state/global`, matches on the backend server `web-pages-01-sv-gstg`, and overrides all values, including the port!\n\nThe result is that we are now attempting to send PROXYv2 traffic to the TLS port.\n\n## The workaround\n\nTo validate the theory and develop a potential workaround, we modify\n`haproxy.cfg` to use a different backend server name.\n\nThe new diff is:\n```diff\n-    server web-pages-01-sv-gstg         web-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080\n-    server web-pages-02-sv-gstg         web-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080\n+    server web-pages-01-sv-gstg-proxyv2 web-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n+    server web-pages-02-sv-gstg-proxyv2 web-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n```\n\nWith this config change in place, we reload HAProxy and indeed, it is now\nserving traffic correctly. See [the merge request fixing it](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/261).\n\n## A follow-up on those `QUIT` bytes\n\nNow, what is up with that `QUIT` message? Is it part of the PROXY protocol? Remember, searching [the\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for that\nstring did not find any matches. However, Matt actually read the spec, and found this section on version 2 of\nthe protocol:\n```\nThe binary header format starts with a constant 12 bytes block containing the\nprotocol signature :\n\n   \\x0D \\x0A \\x0D \\x0A \\x00 \\x0D \\x0A \\x51 \\x55 \\x49 \\x54 \\x0A\n```\n\nThose are indeed the bytes that make up \"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\". Slightly less mnemonic than the header from text-based version 1 of the protocol:\n```\n- a string identifying the protocol : \"PROXY\" ( \\x50 \\x52 \\x4F \\x58 \\x59 )\n  Seeing this string indicates that this is version 1 of the protocol.\n```\n\nWell, I suppose that explains it.\n\nI believe our work here is done. Don't forget to like and subscribe!\n","engineering",[23,24],"production","inside GitLab",{"slug":26,"featured":6,"template":27},"this-sre-attempted-to-roll-out-an-haproxy-change","BlogPost","content:en-us:blog:this-sre-attempted-to-roll-out-an-haproxy-change.yml","yaml","This Sre Attempted To Roll Out An Haproxy Change","content","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change.yml","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","yml",{"_path":36,"_dir":37,"_draft":6,"_partial":6,"_locale":7,"data":38,"_id":443,"_type":29,"title":444,"_source":31,"_file":445,"_stem":446,"_extension":34},"/shared/en-us/main-navigation","en-us",{"logo":39,"freeTrial":44,"sales":49,"login":54,"items":59,"search":389,"minimal":420,"duo":434},{"config":40},{"href":41,"dataGaName":42,"dataGaLocation":43},"/","gitlab logo","header",{"text":45,"config":46},"Get free trial",{"href":47,"dataGaName":48,"dataGaLocation":43},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":50,"config":51},"Talk to sales",{"href":52,"dataGaName":53,"dataGaLocation":43},"/sales/","sales",{"text":55,"config":56},"Sign in",{"href":57,"dataGaName":58,"dataGaLocation":43},"https://gitlab.com/users/sign_in/","sign in",[60,104,200,205,310,370],{"text":61,"config":62,"cards":64,"footer":87},"Platform",{"dataNavLevelOne":63},"platform",[65,71,79],{"title":61,"description":66,"link":67},"The most comprehensive AI-powered DevSecOps Platform",{"text":68,"config":69},"Explore our Platform",{"href":70,"dataGaName":63,"dataGaLocation":43},"/platform/",{"title":72,"description":73,"link":74},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":75,"config":76},"Meet GitLab Duo",{"href":77,"dataGaName":78,"dataGaLocation":43},"/gitlab-duo/","gitlab duo ai",{"title":80,"description":81,"link":82},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":83,"config":84},"Learn more",{"href":85,"dataGaName":86,"dataGaLocation":43},"/why-gitlab/","why gitlab",{"title":88,"items":89},"Get started with",[90,95,100],{"text":91,"config":92},"Platform Engineering",{"href":93,"dataGaName":94,"dataGaLocation":43},"/solutions/platform-engineering/","platform engineering",{"text":96,"config":97},"Developer Experience",{"href":98,"dataGaName":99,"dataGaLocation":43},"/developer-experience/","Developer experience",{"text":101,"config":102},"MLOps",{"href":103,"dataGaName":101,"dataGaLocation":43},"/topics/devops/the-role-of-ai-in-devops/",{"text":105,"left":106,"config":107,"link":109,"lists":113,"footer":182},"Product",true,{"dataNavLevelOne":108},"solutions",{"text":110,"config":111},"View all Solutions",{"href":112,"dataGaName":108,"dataGaLocation":43},"/solutions/",[114,139,161],{"title":115,"description":116,"link":117,"items":122},"Automation","CI/CD and automation to accelerate deployment",{"config":118},{"icon":119,"href":120,"dataGaName":121,"dataGaLocation":43},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[123,127,131,135],{"text":124,"config":125},"CI/CD",{"href":126,"dataGaLocation":43,"dataGaName":124},"/solutions/continuous-integration/",{"text":128,"config":129},"AI-Assisted Development",{"href":77,"dataGaLocation":43,"dataGaName":130},"AI assisted development",{"text":132,"config":133},"Source Code Management",{"href":134,"dataGaLocation":43,"dataGaName":132},"/solutions/source-code-management/",{"text":136,"config":137},"Automated Software Delivery",{"href":120,"dataGaLocation":43,"dataGaName":138},"Automated software delivery",{"title":140,"description":141,"link":142,"items":147},"Security","Deliver code faster without compromising security",{"config":143},{"href":144,"dataGaName":145,"dataGaLocation":43,"icon":146},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[148,151,156],{"text":149,"config":150},"Security & Compliance",{"href":144,"dataGaLocation":43,"dataGaName":149},{"text":152,"config":153},"Software Supply Chain Security",{"href":154,"dataGaLocation":43,"dataGaName":155},"/solutions/supply-chain/","Software supply chain security",{"text":157,"config":158},"Compliance & Governance",{"href":159,"dataGaLocation":43,"dataGaName":160},"/solutions/continuous-software-compliance/","Compliance and governance",{"title":162,"link":163,"items":168},"Measurement",{"config":164},{"icon":165,"href":166,"dataGaName":167,"dataGaLocation":43},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[169,173,177],{"text":170,"config":171},"Visibility & Measurement",{"href":166,"dataGaLocation":43,"dataGaName":172},"Visibility and Measurement",{"text":174,"config":175},"Value Stream Management",{"href":176,"dataGaLocation":43,"dataGaName":174},"/solutions/value-stream-management/",{"text":178,"config":179},"Analytics & Insights",{"href":180,"dataGaLocation":43,"dataGaName":181},"/solutions/analytics-and-insights/","Analytics and insights",{"title":183,"items":184},"GitLab for",[185,190,195],{"text":186,"config":187},"Enterprise",{"href":188,"dataGaLocation":43,"dataGaName":189},"/enterprise/","enterprise",{"text":191,"config":192},"Small Business",{"href":193,"dataGaLocation":43,"dataGaName":194},"/small-business/","small business",{"text":196,"config":197},"Public Sector",{"href":198,"dataGaLocation":43,"dataGaName":199},"/solutions/public-sector/","public sector",{"text":201,"config":202},"Pricing",{"href":203,"dataGaName":204,"dataGaLocation":43,"dataNavLevelOne":204},"/pricing/","pricing",{"text":206,"config":207,"link":209,"lists":213,"feature":297},"Resources",{"dataNavLevelOne":208},"resources",{"text":210,"config":211},"View all resources",{"href":212,"dataGaName":208,"dataGaLocation":43},"/resources/",[214,247,269],{"title":215,"items":216},"Getting started",[217,222,227,232,237,242],{"text":218,"config":219},"Install",{"href":220,"dataGaName":221,"dataGaLocation":43},"/install/","install",{"text":223,"config":224},"Quick start guides",{"href":225,"dataGaName":226,"dataGaLocation":43},"/get-started/","quick setup checklists",{"text":228,"config":229},"Learn",{"href":230,"dataGaLocation":43,"dataGaName":231},"https://university.gitlab.com/","learn",{"text":233,"config":234},"Product documentation",{"href":235,"dataGaName":236,"dataGaLocation":43},"https://docs.gitlab.com/","product documentation",{"text":238,"config":239},"Best practice videos",{"href":240,"dataGaName":241,"dataGaLocation":43},"/getting-started-videos/","best practice videos",{"text":243,"config":244},"Integrations",{"href":245,"dataGaName":246,"dataGaLocation":43},"/integrations/","integrations",{"title":248,"items":249},"Discover",[250,255,259,264],{"text":251,"config":252},"Customer success stories",{"href":253,"dataGaName":254,"dataGaLocation":43},"/customers/","customer success stories",{"text":256,"config":257},"Blog",{"href":258,"dataGaName":5,"dataGaLocation":43},"/blog/",{"text":260,"config":261},"Remote",{"href":262,"dataGaName":263,"dataGaLocation":43},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":265,"config":266},"TeamOps",{"href":267,"dataGaName":268,"dataGaLocation":43},"/teamops/","teamops",{"title":270,"items":271},"Connect",[272,277,282,287,292],{"text":273,"config":274},"GitLab Services",{"href":275,"dataGaName":276,"dataGaLocation":43},"/services/","services",{"text":278,"config":279},"Community",{"href":280,"dataGaName":281,"dataGaLocation":43},"/community/","community",{"text":283,"config":284},"Forum",{"href":285,"dataGaName":286,"dataGaLocation":43},"https://forum.gitlab.com/","forum",{"text":288,"config":289},"Events",{"href":290,"dataGaName":291,"dataGaLocation":43},"/events/","events",{"text":293,"config":294},"Partners",{"href":295,"dataGaName":296,"dataGaLocation":43},"/partners/","partners",{"backgroundColor":298,"textColor":299,"text":300,"image":301,"link":305},"#2f2a6b","#fff","Insights for the future of software development",{"altText":302,"config":303},"the source promo card",{"src":304},"/images/navigation/the-source-promo-card.svg",{"text":306,"config":307},"Read the latest",{"href":308,"dataGaName":309,"dataGaLocation":43},"/the-source/","the source",{"text":311,"config":312,"lists":314},"Company",{"dataNavLevelOne":313},"company",[315],{"items":316},[317,322,328,330,335,340,345,350,355,360,365],{"text":318,"config":319},"About",{"href":320,"dataGaName":321,"dataGaLocation":43},"/company/","about",{"text":323,"config":324,"footerGa":327},"Jobs",{"href":325,"dataGaName":326,"dataGaLocation":43},"/jobs/","jobs",{"dataGaName":326},{"text":288,"config":329},{"href":290,"dataGaName":291,"dataGaLocation":43},{"text":331,"config":332},"Leadership",{"href":333,"dataGaName":334,"dataGaLocation":43},"/company/team/e-group/","leadership",{"text":336,"config":337},"Team",{"href":338,"dataGaName":339,"dataGaLocation":43},"/company/team/","team",{"text":341,"config":342},"Handbook",{"href":343,"dataGaName":344,"dataGaLocation":43},"https://handbook.gitlab.com/","handbook",{"text":346,"config":347},"Investor relations",{"href":348,"dataGaName":349,"dataGaLocation":43},"https://ir.gitlab.com/","investor relations",{"text":351,"config":352},"Trust Center",{"href":353,"dataGaName":354,"dataGaLocation":43},"/security/","trust center",{"text":356,"config":357},"AI Transparency Center",{"href":358,"dataGaName":359,"dataGaLocation":43},"/ai-transparency-center/","ai transparency center",{"text":361,"config":362},"Newsletter",{"href":363,"dataGaName":364,"dataGaLocation":43},"/company/contact/","newsletter",{"text":366,"config":367},"Press",{"href":368,"dataGaName":369,"dataGaLocation":43},"/press/","press",{"text":371,"config":372,"lists":373},"Contact us",{"dataNavLevelOne":313},[374],{"items":375},[376,379,384],{"text":50,"config":377},{"href":52,"dataGaName":378,"dataGaLocation":43},"talk to sales",{"text":380,"config":381},"Get help",{"href":382,"dataGaName":383,"dataGaLocation":43},"/support/","get help",{"text":385,"config":386},"Customer portal",{"href":387,"dataGaName":388,"dataGaLocation":43},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":390,"login":391,"suggestions":398},"Close",{"text":392,"link":393},"To search repositories and projects, login to",{"text":394,"config":395},"gitlab.com",{"href":57,"dataGaName":396,"dataGaLocation":397},"search login","search",{"text":399,"default":400},"Suggestions",[401,403,407,409,413,417],{"text":72,"config":402},{"href":77,"dataGaName":72,"dataGaLocation":397},{"text":404,"config":405},"Code Suggestions (AI)",{"href":406,"dataGaName":404,"dataGaLocation":397},"/solutions/code-suggestions/",{"text":124,"config":408},{"href":126,"dataGaName":124,"dataGaLocation":397},{"text":410,"config":411},"GitLab on AWS",{"href":412,"dataGaName":410,"dataGaLocation":397},"/partners/technology-partners/aws/",{"text":414,"config":415},"GitLab on Google Cloud",{"href":416,"dataGaName":414,"dataGaLocation":397},"/partners/technology-partners/google-cloud-platform/",{"text":418,"config":419},"Why GitLab?",{"href":85,"dataGaName":418,"dataGaLocation":397},{"freeTrial":421,"mobileIcon":426,"desktopIcon":431},{"text":422,"config":423},"Start free trial",{"href":424,"dataGaName":48,"dataGaLocation":425},"https://gitlab.com/-/trials/new/","nav",{"altText":427,"config":428},"Gitlab Icon",{"src":429,"dataGaName":430,"dataGaLocation":425},"/images/brand/gitlab-logo-tanuki.svg","gitlab icon",{"altText":427,"config":432},{"src":433,"dataGaName":430,"dataGaLocation":425},"/images/brand/gitlab-logo-type.svg",{"freeTrial":435,"mobileIcon":439,"desktopIcon":441},{"text":436,"config":437},"Learn more about GitLab Duo",{"href":77,"dataGaName":438,"dataGaLocation":425},"gitlab duo",{"altText":427,"config":440},{"src":429,"dataGaName":430,"dataGaLocation":425},{"altText":427,"config":442},{"src":433,"dataGaName":430,"dataGaLocation":425},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":448,"_dir":37,"_draft":6,"_partial":6,"_locale":7,"title":449,"titleMobile":449,"button":450,"config":455,"_id":457,"_type":29,"_source":31,"_file":458,"_stem":459,"_extension":34},"/shared/en-us/banner","GitLab 18 & the next step in intelligent DevSecOps. Join us June 24.",{"text":451,"config":452},"Register now",{"href":453,"dataGaName":454,"dataGaLocation":43},"/eighteen/","gitlab 18 banner",{"layout":456},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":461,"_dir":37,"_draft":6,"_partial":6,"_locale":7,"data":462,"_id":668,"_type":29,"title":669,"_source":31,"_file":670,"_stem":671,"_extension":34},"/shared/en-us/main-footer",{"text":463,"source":464,"edit":470,"contribute":475,"config":480,"items":485,"minimal":660},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":465,"config":466},"View page source",{"href":467,"dataGaName":468,"dataGaLocation":469},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":471,"config":472},"Edit this page",{"href":473,"dataGaName":474,"dataGaLocation":469},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":476,"config":477},"Please contribute",{"href":478,"dataGaName":479,"dataGaLocation":469},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":481,"facebook":482,"youtube":483,"linkedin":484},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[486,509,566,595,630],{"title":61,"links":487,"subMenu":492},[488],{"text":489,"config":490},"DevSecOps platform",{"href":70,"dataGaName":491,"dataGaLocation":469},"devsecops platform",[493],{"title":201,"links":494},[495,499,504],{"text":496,"config":497},"View plans",{"href":203,"dataGaName":498,"dataGaLocation":469},"view plans",{"text":500,"config":501},"Why Premium?",{"href":502,"dataGaName":503,"dataGaLocation":469},"/pricing/premium/","why premium",{"text":505,"config":506},"Why Ultimate?",{"href":507,"dataGaName":508,"dataGaLocation":469},"/pricing/ultimate/","why ultimate",{"title":510,"links":511},"Solutions",[512,517,520,522,527,532,536,539,543,548,550,553,556,561],{"text":513,"config":514},"Digital transformation",{"href":515,"dataGaName":516,"dataGaLocation":469},"/solutions/digital-transformation/","digital transformation",{"text":149,"config":518},{"href":144,"dataGaName":519,"dataGaLocation":469},"security & compliance",{"text":138,"config":521},{"href":120,"dataGaName":121,"dataGaLocation":469},{"text":523,"config":524},"Agile development",{"href":525,"dataGaName":526,"dataGaLocation":469},"/solutions/agile-delivery/","agile delivery",{"text":528,"config":529},"Cloud transformation",{"href":530,"dataGaName":531,"dataGaLocation":469},"/solutions/cloud-native/","cloud transformation",{"text":533,"config":534},"SCM",{"href":134,"dataGaName":535,"dataGaLocation":469},"source code management",{"text":124,"config":537},{"href":126,"dataGaName":538,"dataGaLocation":469},"continuous integration & delivery",{"text":540,"config":541},"Value stream management",{"href":176,"dataGaName":542,"dataGaLocation":469},"value stream management",{"text":544,"config":545},"GitOps",{"href":546,"dataGaName":547,"dataGaLocation":469},"/solutions/gitops/","gitops",{"text":186,"config":549},{"href":188,"dataGaName":189,"dataGaLocation":469},{"text":551,"config":552},"Small business",{"href":193,"dataGaName":194,"dataGaLocation":469},{"text":554,"config":555},"Public sector",{"href":198,"dataGaName":199,"dataGaLocation":469},{"text":557,"config":558},"Education",{"href":559,"dataGaName":560,"dataGaLocation":469},"/solutions/education/","education",{"text":562,"config":563},"Financial services",{"href":564,"dataGaName":565,"dataGaLocation":469},"/solutions/finance/","financial services",{"title":206,"links":567},[568,570,572,574,577,579,581,583,585,587,589,591,593],{"text":218,"config":569},{"href":220,"dataGaName":221,"dataGaLocation":469},{"text":223,"config":571},{"href":225,"dataGaName":226,"dataGaLocation":469},{"text":228,"config":573},{"href":230,"dataGaName":231,"dataGaLocation":469},{"text":233,"config":575},{"href":235,"dataGaName":576,"dataGaLocation":469},"docs",{"text":256,"config":578},{"href":258,"dataGaName":5,"dataGaLocation":469},{"text":251,"config":580},{"href":253,"dataGaName":254,"dataGaLocation":469},{"text":260,"config":582},{"href":262,"dataGaName":263,"dataGaLocation":469},{"text":273,"config":584},{"href":275,"dataGaName":276,"dataGaLocation":469},{"text":265,"config":586},{"href":267,"dataGaName":268,"dataGaLocation":469},{"text":278,"config":588},{"href":280,"dataGaName":281,"dataGaLocation":469},{"text":283,"config":590},{"href":285,"dataGaName":286,"dataGaLocation":469},{"text":288,"config":592},{"href":290,"dataGaName":291,"dataGaLocation":469},{"text":293,"config":594},{"href":295,"dataGaName":296,"dataGaLocation":469},{"title":311,"links":596},[597,599,601,603,605,607,609,614,619,621,623,625],{"text":318,"config":598},{"href":320,"dataGaName":313,"dataGaLocation":469},{"text":323,"config":600},{"href":325,"dataGaName":326,"dataGaLocation":469},{"text":331,"config":602},{"href":333,"dataGaName":334,"dataGaLocation":469},{"text":336,"config":604},{"href":338,"dataGaName":339,"dataGaLocation":469},{"text":341,"config":606},{"href":343,"dataGaName":344,"dataGaLocation":469},{"text":346,"config":608},{"href":348,"dataGaName":349,"dataGaLocation":469},{"text":610,"config":611},"Environmental, social and governance (ESG)",{"href":612,"dataGaName":613,"dataGaLocation":469},"/environmental-social-governance/","environmental, social and governance",{"text":615,"config":616},"Diversity, inclusion and belonging (DIB)",{"href":617,"dataGaName":618,"dataGaLocation":469},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":351,"config":620},{"href":353,"dataGaName":354,"dataGaLocation":469},{"text":361,"config":622},{"href":363,"dataGaName":364,"dataGaLocation":469},{"text":366,"config":624},{"href":368,"dataGaName":369,"dataGaLocation":469},{"text":626,"config":627},"Modern Slavery Transparency Statement",{"href":628,"dataGaName":629,"dataGaLocation":469},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":631,"links":632},"Contact Us",[633,636,638,640,645,650,655],{"text":634,"config":635},"Contact an expert",{"href":52,"dataGaName":53,"dataGaLocation":469},{"text":380,"config":637},{"href":382,"dataGaName":383,"dataGaLocation":469},{"text":385,"config":639},{"href":387,"dataGaName":388,"dataGaLocation":469},{"text":641,"config":642},"Status",{"href":643,"dataGaName":644,"dataGaLocation":469},"https://status.gitlab.com/","status",{"text":646,"config":647},"Terms of use",{"href":648,"dataGaName":649,"dataGaLocation":469},"/terms/","terms of use",{"text":651,"config":652},"Privacy statement",{"href":653,"dataGaName":654,"dataGaLocation":469},"/privacy/","privacy statement",{"text":656,"config":657},"Cookie preferences",{"dataGaName":658,"dataGaLocation":469,"id":659,"isOneTrustButton":106},"cookie preferences","ot-sdk-btn",{"items":661},[662,664,666],{"text":646,"config":663},{"href":648,"dataGaName":649,"dataGaLocation":469},{"text":651,"config":665},{"href":653,"dataGaName":654,"dataGaLocation":469},{"text":656,"config":667},{"dataGaName":658,"dataGaLocation":469,"id":659,"isOneTrustButton":106},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[673],{"_path":674,"_dir":675,"_draft":6,"_partial":6,"_locale":7,"content":676,"config":680,"_id":682,"_type":29,"title":18,"_source":31,"_file":683,"_stem":684,"_extension":34},"/en-us/blog/authors/igor-wiedler","authors",{"name":18,"config":677},{"headshot":678,"ctfId":679},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681841/Blog/Author%20Headshots/igorwwwwwwwwwwwwwwwwwwww-headshot.png","igorwwwwwwwwwwwwwwwwwwww",{"template":681},"BlogAuthor","content:en-us:blog:authors:igor-wiedler.yml","en-us/blog/authors/igor-wiedler.yml","en-us/blog/authors/igor-wiedler",{"_path":686,"_dir":37,"_draft":6,"_partial":6,"_locale":7,"header":687,"eyebrow":688,"blurb":689,"button":690,"secondaryButton":694,"_id":696,"_type":29,"title":697,"_source":31,"_file":698,"_stem":699,"_extension":34},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":45,"config":691},{"href":692,"dataGaName":48,"dataGaLocation":693},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":50,"config":695},{"href":52,"dataGaName":53,"dataGaLocation":693},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1751484603701]