Tristan Rice

Software Engineer and Student

Running Untrusted Code in a Secure Docker Container from Scratch

4 Minutes 800 Words

As part of Luk.ai we need to be able to run Tensorflow within a secure environment since a running Tensorflow model can do pretty much anything it wants to the host system.

For ease of deployment, we’d also like to be able to use Docker since it provides nice sandboxing support and ability to limit resources used by the container. We’d also like for the container to not be able to do anything other than run models. Thus, we needed a completely stripped down container with no permissions set. It’s also pretty nice if it’s as small as possible.

Building A Minimal Container

Thus enters building a Dockerfile with just the minimum dependencies.

Normally you can just build a Go binary without any shared dependencies by running:

$ go build -v -tags netgo -installsuffix netgo -ldflags '-w -s' .

However, since this container needs to be able to run Tensorflow we need to be able to link against the libtensorflow.so file and it’s dependencies. To find all the dependencies we can use ldd.

$ ldd secagg
        linux-vdso.so.1 (0x00007ffcbd1c3000)
        libtensorflow.so => /usr/lib/libtensorflow.so (0x00007f007ab54000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f007a936000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f007a57a000)
        libtensorflow_framework.so => /usr/lib/libtensorflow_framework.so (0x00007f0079590000)
        libcublas.so.9.1 => /opt/cuda/lib64/libcublas.so.9.1 (0x00007f0075e6c000)
        libcusolver.so.9.1 => /opt/cuda/lib64/libcusolver.so.9.1 (0x00007f00706f7000)
        libcudart.so.9.1 => /opt/cuda/lib64/libcudart.so.9.1 (0x00007f0070489000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f0070285000)
        libgomp.so.1 => /usr/lib/libgomp.so.1 (0x00007f0070057000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f006fcc2000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f006f939000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f006f721000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f00a0484000)
        libcuda.so.1 => /usr/lib/libcuda.so.1 (0x00007f006eb81000)
        libcudnn.so.7 => /opt/cuda/lib64/libcudnn.so.7 (0x00007f005a31e000)
        libcufft.so.9.1 => /opt/cuda/lib64/libcufft.so.9.1 (0x00007f0052e31000)
        libcurand.so.9.1 => /opt/cuda/lib64/libcurand.so.9.1 (0x00007f004eeae000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f004eca6000)
        libnvidia-fatbinaryloader.so.390.48 => /usr/lib/libnvidia-fatbinaryloader.so.390.48 (0x00007f004ea5a000

The linux-vdso.so.1 file isn’t a real dependency and is automatically injected by the Linux kernel, the rest however we need to include in the container.

There’s a little bit of bash code used to copy all of those files into a directory root/.

mkdir -p root
cd root
for f in $(ldd ../secagg | sed -n 's/.*\s\(\/.*\) .*/\1/p'); do
  cp --parents "$f" .
done
cd ..

There’s still one other requirement to get a running system and that’s a /etc/passwd file for the nobody user.

passwd.minimal

nobody:x:65534:65534:Nobody:/:

Once we have all that, we can now create the Dockerfile and build the container.

FROM scratch
ENV LD_LIBRARY_PATH /usr/local/lib:/usr/lib:/lib:/opt/cuda/lib64:/usr/lib64:/lib64
ADD root /
ADD passwd.minimal /etc/passwd
ADD secagg /secagg
USER nobody
CMD ["/usr/lib64/ld-linux-x86-64.so.2", "/secagg"]

LD_LIBRARY_PATH

Docker will automatically detect shared files under /lib and /lib64 but if they’re anywhere else we need to tell the system where to load them from. You can do this by setting the LD_LIBRARY_PATH environment variable and then using ld.so to run the binary.

The LD_LIBRARY_PATH above is what we needed to run the container on Arch Linux, but it’ll depend on what operating system it’s being built on.

If you get a docker error like standard_init_linux.go:190: exec user process caused "no such file or directory" it’s likely a dynamic linking issue.

Running the Container

To actually run the container, we launch it from the Go host process via the Docker command line.

ctx, cancel := context.WithCancel(context.TODO())
cmd := exec.CommandContext(
	ctx,
	"docker", "run",
	fmt.Sprintf("--name=%s", container),
	// disable network access
	"--net=none",
	// drop all process capabilities
	"--cap-drop=all",
	// attach to STDIN, STDOUT, STDERR
	"--attach=STDIN",
	"--attach=STDOUT",
	"--attach=STDERR",
	// allow sending and receiving on STDIN
	"-i",
	// limit CPU usage to 1 core
	"--cpus=1",
	// remove container after shutting down
	"--rm",
	// make the root file system read only
	"--read-only",
	// setup a /tmp directory without execute permission and limit to 1GB
	"--tmpfs=/tmp:rw,size=1g,mode=1777,noexec",
	"secagg:latest",
)
cmd.Stderr = os.Stderr
in, err := cmd.StdinPipe()
if err != nil {
	return nil, err
}
out, err := cmd.StdoutPipe()
if err != nil {
	return nil, err
}
if err := cmd.Start(); err != nil {
  return nil, err
}

Communicating with the Container

Since the container is completely locked down with no network access we use the stdin and stdout for communication. Logs are sent over stderr like normal.

Since we like having strongly typed messages, we use GRPC on both sides along with muxado so we can have multiple connections to the container running over the same ReadWriteCloser.

Client/Host Side

var _ io.ReadWriteCloser = RWC{}

var StdIORW = RWC{
	ReadCloser:  os.Stdin,
	WriteCloser: os.Stdout,
}

type RWC struct {
	io.ReadCloser
	io.WriteCloser
}

func (rw RWC) Close() error {
	if err := rw.WriteCloser.Close(); err != nil {
		return err
	}
	if err := rw.ReadCloser.Close(); err != nil {
		return err
	}
	return nil
}

// code to setup the connection over the stdin/stdout writer and reader.
rwc := seclib.RWC{
	ReadCloser:  out,
	WriteCloser: in,
}
mux := muxado.Client(rwc, nil)
conn, err := grpc.Dial(
	"",
	grpc.WithDialer(func(addr string, dur time.Duration) (gonet.Conn, error) {
		return mux.Open()
	}),
	grpc.WithInsecure(),
)

Server/Container Side

func main() {
	log.SetPrefix("[SecAgg] ")
	log.SetOutput(os.Stderr)
	log.SetFlags(log.Flags() | log.Lshortfile)

	log.Println("Running secure aggregator!")

	s := newRPCServer()
	grpcServer := grpc.NewServer(logging.GRPCCallLogger()...)
	secaggpb.RegisterAggServer(grpcServer, s)
	mux := muxado.Server(StdIORW, nil)
	defer mux.Close()
	if err := grpcServer.Serve(mux); err != nil {
		log.Fatal(err)
	}
	log.Println("Done!")
}