This talk is about a project I worked on in the
past two year for which I received some funding from the NLnet Foundation as part of the NGI Zero
Core program.
The project is about trying to improve the live-bootstrap project. Before I go into the details, lets first start
with a demo.
Demo
Lets start with a demo.
./task5_amd64.sh
Result: The program tcc
64-bits executable for Linux on Intel
Tiny C Compiler 0.9.27
It is build from 'scratch'
The demo is about running the script task5_amd64.sh, which finishes in about half a minute and
produces the tcc program. This is a 64-bits executable in the
Executable and Linkable Format (ELF) for Linux on Intel processors. The
program is build from the Tiny C Compiler 0.9.27 sources, which are written in
the C programming language itself. TCC is a rather complete, but not so tiny,
compiler for the C programming language and written in the C programming
itself.
The compiler is build from 'scrath'. Later in the talks, I will explain what
that exactly means.
Why is this fast?
Original for 32-bits: 11 minutes
For 64-bits: Fails after 13 minutes
GNU Mes C compiler
Building this with the original live-bootstrap takes 11 minutes and 38 seconds on my laptop when building
for 32-bits Intel target (also known as x86). Trying to build it for the for
64-bits Intel target (also known AMD64) fails after 13 minutes and 38 seconds
because some problem with the TCC 0.9.26 code.
The 64-bits target is not yet supported by the live-bootstrap project.
The main reason why it is slow is because of the GNU Mes compiler.
GNU Mes C compiler
Complete C compiler
Written in Scheme
Scheme interpreter in subset of C
Interpreting parser?
Needs a subset of C compiler
A rather convoluted solution
The GNU Mes C compiler is a
rather complete C compiler that allows you to compile separate C files and
link the object files into an executable. The compiler is written in
Scheme (a dialect in the LISP family). The Scheme program is interpreted
by a Scheme interpreter written in a subset of C. It might be the case that
the compiler written in Scheme makes use of an interpreting parser. (The
C compiler written in Scheme can compile the Scheme interpreter written in C.)
It is also slow because it compiles many individual C files into object files,
which are linked at the end. Every time all the Scheme files are loaded before
the compiler can start its actual work.
This all makes for a rather convoluted solution. This made me look for a
possibility to remove the GNU Mes compiler.
MES replacement project
Find the simplest C compiler
Minimal feature support
No focus on speed
Compile only a single C program
The idea of the MES-replacement project was to write the simplest possible C compiler for
compiling the Tiny C Compiler sources and the other tools that are needed to
unzip and untar the source repository. The compiler will presume that the input
is correct. So, almost no error messages and very limited type checking. The
Tiny C Compiler can be compiled as a single file that includes all other
files, which means you only be able to compile a single C file to an
executable.
The idea was to write a compiler that would generate code for a simple
stack based language geared to C. Then write a very simple compiler to
compile this to assembly and next use the M1 and hex2 tools from the
original live-bootstrap project to compile it to an executable ELF file.
The C compiler is the tcc_cc program that produces output for the Stack-C language, which can be compiled to assembly with the stack_c program. The assembly output is then with the help of
original live-bootstrap programs M1, blood-elf, and
hex2 compiled into an ELF executable.
I first implemented the tool chain for 32-bits intel (also known as x86).
Next I worked on a tool chain for 64-bits (also know as AM64), which took a
bit more time debugging then I had expected. I got some help from
Harmen Stoppels for fixing some
problems with TCC sources. Executing the task5_amd64.sh now took one
minute and 22 seconds.
Harmen also helped me with the Aarch64 implementation using some AI.
Harmen helped because he wanted to use it for his shpack, a fast,
bootstrappalble package manager for Limux. He is working in the High
Performance Computing community where there sometimes is a need to
try out combinations of compilers and standard libraries for building Linux.
shpack is using musl instead
of the GNU Mes standard library.
Performance
The program hex2 is rather slow
How does Linux run a program?
Segmentation error
System interupts:
Ask more memory
Open file
Read from file
Write to file
...
Standard library
Reading and writing single characters
Every memory allocation
Implement buffering
Harmen implemented some further speed improvements in the code he used from
the MES-replacement project. This made me look at my implementationa again.
I had already noticed that the program hex2 took the longest time to
execute.
When Linux starts a program it encloses that program in its own part of
memory. If program tries to access memory outside of the allocated range,
Linux quits the program with the famouse 'Segmentation error'. If a program
want to communicate with the outaide it has to do this through a system
interupt.
The C programs do not directly call system interupts but do this through the
so-called Standard library. The MES replacement project uses its own
standard library. To keep it as simple as possible, it is not very efficient.
It issues system interupt for reading and writing single characters and for
each allocation of memory.
I implemented some buffering for these operations to reduce the number of
system calls and that reduced the execution time of the script by more than
half.
In 1984, Ken Thompson
gave his Turing Award acceptance speech Reflections on Trusting Trust. In this he demonstrated the
possibility of a malicious compiler that when used for building an operating
systems could introduce a backdoor into the operating systems. Such a compiler
could recognise when it is compiled by itself and duplicate the malicious
code in the resulting compiler, making it malicious as well.
Bootstrapping stands for the impossible idea of pulling yourself up from your
bootstraps. This concept in computer science can be translated as starting from scratch.
If you want to execute something on a computer you need some kind of binary
program. The idea is to start with the smallest possible program and use that
to build other programs that could be used to again build more advanced
programs until you can build a highlevel compiler to compile everything from
source.
The left bar of the T shape represents the inputs of the program, the
right bar the output of the program and the bottom bar the program itself.
Usually the type of the input, the output and the program is shown inside
the T shape.
The output can only be trusted when both the input and the program can be
trusted.