As security researchers, the GitHub Security Lab team constantly embarks
on an emotional journey with each new vulnerability challenge.
The excitement of starting new research, the disappointment that comes
with hitting a plateau, the energy it takes to stay focused and on target…
and hopefully, the sheer joy of achieving a tangible result
after weeks or months of working on a problem that seemed unsolvable.
Regardless of how proud you are of the results, do you ever get
a nagging feeling that maybe you didn’t make enough of an impact?
While single bug fixes are worthwhile in improving code,
it’s not sufficient enough to improve the state of security
of the open source software (OSS) ecosystem as a whole.
This holds true especially when you consider that
software is always growing and changing?and as vulnerabilities are fixed,
new ones are introduced.
At GitHub, we host millions of OSS projects which puts us in a unique position
to take a different approach with OSS security.
We have the power and responsibility to make an impact beyond single bug fixes.
This is why a big part of the GitHub Security Lab mission is to find ways
to scale our vulnerability hunting efforts and empower others to do the same.
Our goal is to turn single vulnerabilities into hundreds, if not thousands,
of bug fixes at a time. Enabled by the GitHub engineering teams,
we aim to establish workflows that are open to the community that
tackle vulnerabilities at scale on the GitHub platform.
Ultimately, we want to establish feedback loops with the developer
and security communities, and act as security facilitators,
all while working with the OSS community to secure the software we all depend upon.
GitHub and CERT
Dial up modem sounds are a thing of the past, but the protocols we used
while listening to a modem’s cheerful
sounds
are still present and active all over the internet.
Nostalgia aside, things got interesting when GitHub Security Lab
received an email from Vijay Sarvepalli from the
CERT team
informing us that
IOActive
’s Ilja Van Sprundel
found a remote
vulnerability
in pppd (point-to-point protocol daemon).
When we investigated the unpatched pppd Extensible Authentication Protocol (EAP) code,
we noted the following construct:
/* Not so likely to happen. */
if
(
vallen
>=
len
+
sizeof
(
rhostname
))
{
ppp_dbglog
((
"EAP: trimming really long peer name down"
));
MEMCPY
(
rhostname
,
inp
+
vallen
,
sizeof
(
rhostname
)
-
1
);
rhostname
[
sizeof
(
rhostname
)
-
1
]
=
'\0'
;
Given that previously
vallen
was checked to be less than
len
,
the vulnerability occurs because it can never be the case
that
vallen >= len + sizeof(rhostname)
.
Therefore,
rhostname
never gets trimmed against oversized input,
and the
rhostname
array may overflow.
At the Security Lab, we love to learn about new vulnerabilities
so we can model them in CodeQL. CodeQL is a special querying language
that allows you to query code as data to find instances of vulnerabilities.
However, the real reason the CERT contacted us was to help patch
thousands of GitHub projects that were using this exact same vulnerable code pattern.
More information about this could be found on the
CERT blog post
and
CVE
.
This includes various projects, such as complete Android forks
to
robot-soccer projects
.
We’re always extremely cautious when it comes to sending mass project notifications at GitHub.
While we prefer not to disturb maintainers, this vulnerability was public,
potentially remotely exploitable, and a significant amount of these projects
were most likely unaware of its existence.
We ultimately decided to create a mass pull request aimed at fixing this vulnerability
in as many projects as possible. This pull request automatically proposes minor changes
to project code for all affected projects, and can be applied by maintainers using a few clicks.
After confirmation from GitHub leadership about moving forward,
our very own
@pwntester
embarked on the implementation.
First, he collected potentially vulnerable OSS projects using GitHub code search,
followed by trimming duplicates. After this initial process,
he used a customized version of a script that community member
Jonathan Leitschuh
developed for a similar mass security pull request effort.
We’ll cover more about this pull request later in this post.
With a bit of API juggling, script tweaking, and some patience,
we started the process of creating pull requests against a large number of vulnerable GitHub projects.
To date
1,964 different projects have received this security pull request
and in the first 72 hours, 45 have actually merged the patch
.
Some of the projects that merged the pull request include:
- NanoTube
: A lwIP-based networking library for the TI-84 Plus CE
- Netgear-D7800-Openwrt
: Openwrt support for the Netgear D7800 router
- AliOS
: The Alibaba Operating System
Monitor the progress of this mass pull request
GitHub and the community
As you might know, the Security Lab runs a
bug bounty program
.
It’s different from other bounty programs in that we don’t award unique vulnerability findings,
but rather novel CodeQL queries that model vulnerability patterns that the CodeQL team
can include in our standard query pack. Once a CodeQL query makes it into our standard query pack,
it’s run on all open source projects for free, thus amplifying its impact.
The other bounty we run encourages the community to write CodeQL queries
that help find and report more than three vulnerabilities in OSS projects.
Ultimately, the goal is to scale and move away from the individual fix model
of traditional security patching.
These bounties have made a great impact to date, and one of our favorite stories
from a community member
Jonathan Leitschuh
who submitted a
CodeQL query for detecting insecure Maven Java artifact
downloads that were susceptible to person-in-the-middle tampering.
He received a bounty for that query, but then he decided to use the query to actually fix this problem at scale.
He wrote a bot that opened over
1,700
pull requests for any affected GitHub projects.
import java
import semmle.code.xml.MavenPom
private class DeclaredRepository extends PomElement {
DeclaredRepository() {
this.getName() = "repository" or
this.getName() = "snapshotRepository" or
this.getName() = "pluginRepository"
}
string getUrl() { result = getAChild("url").(PomElement).getValue() }
predicate isInsecureRepositoryUsage () {
getUrl().regexpMatch("(?i)^(http|ftp)://(?!localhost[:/]).*")
}
}
from DeclaredRepository repository
where repository.isInsecureRepositoryUsage()
select repository,
"Downloading or uploading artifacts over insecure protocol (e.g. http or ftp) to/from repository " +
repository.getUrl()
Jonathan’s success with fixing this issue across hundreds of projects partially inspired the workflow we adopted
to coordinate our own mass pull request effort with CERT.
It’s a great example of the kind of feedback loop between us and the community
that we want to further develop and enable. We love to see the community going the extra mile,
which is why we ended up doubling Jonathan’s
bounty payment
as a token of appreciation for his continued collaboration with our team and the wider community.
Going forward
As a security research team, it’s in our DNA to keep trying no matter how insurmountable
the OSS security challenge may seem. Sometimes we’ll succeed and often we’ll fail,
but we always walk away from the situation with more insight and knowledge.
We hope to iteratively improve our success rates and apply the concept of embracing failure
as we know it’s a fundamental ingredient of growth to software security itself.
Working closely with the wider GitHub organization, we hope to produce workflows
that enable us and our community to make GitHub the home for open source security.
If coding is social, then security should be social too.
As always, we’d love to hear about your unique security challenges,
from code-review to coordinated disclosure, and everything in between.
If there’s anything we can help you with, reach out to the
@GHSecurityLab
team.
You can also
get involved
and help us secure open source at scale!
Visit our
bounty program
, we look forward to your submissions.