Archive

Archive for décembre 2012

2012 in review

31 décembre 2012 1 commentaire

Les lutins statisticiens chez WordPress.com ont préparé un rapport annuel 2012 pour ce blog.

Cliquez ici pour voir le rapport complet.

Publicités
Catégories :Uncategorized

A minimal linux system – Part 3

28 décembre 2012 Laisser un commentaire

We have managed, using only our bare hands, to :
– build a gcc-based cross-compiler
– build a linux kernel binary
– build a C standard library

Now we need some executables for our system. A minimum set would include a shell interpreter, some basic commands (cat, grep, ls, cp and so on) and an ssh server, because ssh is great.

The most current implementation of these tools on ‘desktop’ linux distributions are provided by GNU coreutils, GNU bash and openssh.

busybox

As more convenient way is to use the busybox project : busybox combines a shell interpreter all all the needed commonly used programs into a single binary, detecting how to behave depending on the name that was used to invoke it.

As usual it is simple to build, as long as you give the good configuration in the .config file (use mine or make menuconfig !)

BUSYBOX_VERSION=1.20.2
pushd $TOOLCHAIN_INSTALL_DIR/src
curl "http://www.busybox.net/downloads/busybox-$BUSYBOX_VERSION.tar.bz2" -o busybox-$BUSYBOX_VERSION.tar.bz2
tar -jxvf busybox-$BUSYBOX_VERSION.tar.bz2
ln -s busybox-$BUSYBOX_VERSION.tar.bz2 busybox
pushd busybox
    #edit .config
    make "ARCH=$ARCH" "CROSS_COMPILE=$TOOLCHAIN_TARGET-" "PREFIX=$TOOLCHAIN_INSTALL_DIR" CFLAGS="-I$TOOLCHAIN_INSTALL_DIR/include -Os -fPIC"
popd
popd

zlib

The zlib compression library is really common and offently used, so I decided to embed it as a shared library.

ZLIB_VERSION=1.2.7
pushd $TOOLCHAIN_INSTALL_DIR/src
curl "http://zlib.net/zlib-$ZLIB_VERSION.tar.gz" -o zlib-$ZLIB_VERSION.tar.gz
tar -zxvf zlib-$ZLIB_VERSION.tar.gz
ln -s zlib-$ZLIB_VERSION zlib
pushd zlib
	export CC=$TOOLCHAIN_TARGET-gcc
	export CFLAGS="-Os -I$TOOLCHAIN_INSTALL_DIR/include"
	export LDSHARED="$TOOLCHAIN_TARGET-gcc -shared -Wl,-soname,libz.so.1"
	./configure --shared --prefix=/usr/$TOOLCHAIN_TARGET
	make -B
	make install prefix=$TOOLCHAIN_INSTALL_DIR/usr/$TOOLCHAIN_TARGET
popd
popd

dropbear

dropbear is a super lightweight implementation of the ssh2 protocol, which make it the perfect alternative to openssh ! it compiles without any dependency and word pretty well – just like busybox, several executables are bundled in the same binary. I choosed to include the ssh server, the client, and scp.

DROPBEAR_VERSION=2012.55

pushd $TOOLCHAIN_INSTALL_DIR/src

curl "https://matt.ucc.asn.au/dropbear/dropbear-$DROPBEAR_VERSION.tar.bz2" -o dropbear-$DROPBEAR_VERSION.tar.bz2
tar -jxvf dropbear-$DROPBEAR_VERSION.tar.bz2
ln -s dropbear-$DROPBEAR_VERSION dropbear
pushd dropbear
	export CC=$TOOLCHAIN_TARGET-gcc
	export CFLAGS=-Os

	export C_INCLUDE_PATH=$TOOLCHAIN_INSTALL_DIR/include
	./configure --host=$TOOLCHAIN_TARGET \

		--with-zlib=$TOOLCHAIN_INSTALL_DIR/usr/$TOOLCHAIN_TARGET/lib
	make MULTI=1 PROGRAMS="dropbear scp dbclient"
popd
popd

Now we have all the ingredients : kernel, libraries, executables. on next episode I’ll try to explain how integrate all of these in order to get a functional system – A part that I found more more tricky than compiling the binaries themselves…

Catégories :Uncategorized

A minimal linux system – Part 2

28 décembre 2012 Laisser un commentaire

Ok, we have a cross-compiler toolchain. now we can deal with the real thing. At this point, I was tempted to compile standard C and C++ standard libraries – But this is simply not possible without the header files provided .. by the linux kernel.
It is quite logical because implementations of standard C functions rely on system calls (it seems stupid once said) – So one cannot simply compile a standard library without kernel binaries, that’s why it is called a kernel after all.

so the most dramatic moment of this tutorial begins now : building the linux kernel itself.

Linux kernel

ARCH=x86
LINUX_KERNEL_VERSION=3.6.11

mkdir -p $TOOLCHAIN_INSTALL_DIR/src
pushd $TOOLCHAIN_INSTALL_DIR/src
curl "http://www.kernel.org/pub/linux/kernel/v3.0/linux-$LINUX_KERNEL_VERSION.tar.bz2" -o linux-$LINUX_KERNEL_VERSION.tar.bz2
tar -jxvf linux-$LINUX_KERNEL_VERSION.tar.bz2
ln -s linux-$LINUX_KERNEL_VERSION linux
pushd linux

#linux kernel compilation options is normally setup by editing a .config file at the root of the linux source tree.
#This configuration file may be easily modified by doing :
# apt-get install libncurses5-dev
# make menuconfig
#
# I have my own settings right in this file, so
curl "http://julien-rialland.fr/pub/linux/linux-x86-config" -o .config

#At this step I can apply various kernel patches, for example Prempt-RT (http://en.wikipedia.org/wiki/RTLinux)
#curl "http://www.kernel.org/pub/linux/kernel/projects/rt/3.6/patch-3.6.11-rt25.patch.bz2" -o patch-3.6.11-rt25.patch.bz2
#cat patch-3.6.11-rt25.patch.bz2 | bunzip2 | patch -p1

make "ARCH=$ARCH" "CROSS_COMPILE=$TOOLCHAIN_TARGET-" bzImage
make "ARCH=$ARCH" "INSTALL_HDR_PATH=$TOOLCHAIN_INSTALL_DIR" headers_install

#we also copy some headers that will be needed by libc headers
for h in `find ./include/asm-generic -name "*.h"`; do
    name=`basename $h`
    if [ ! -f "$TOOLCHAIN_INSTALL_DIR/include/asm/$name" ]; then
        ln -s $h $TOOLCHAIN_INSTALL_DIR/include/asm/$name
    fi
done

popd
popd

standard C library : uclibc

On a decent platform, executable are linked with a standard library, which provides implementation of all standard C functions (no ‘hello world’ without it 🙂 )
We use our freshly-compiled compiler in order to compile it (uh, a lot of compilations don’t you think ?)
There are several implementation of the standard C library : the GNU libc (which may be quite bloated for my use), newlib, dietlibc, musl, uclibc. I choosed uclibc because I have no imagination, but I think that any of the listed ones would be fine.

UCLIBC_VERSION=0.9.33.1
push $TOOLCHAIN_INSTALL_DIR/src
curl "http://www.uclibc.org/downloads/uClibc-$UCLIBC_VERSION.tar.bz2" -o uClibc-$UCLIBC_VERSION.tar.bz2
tar -jxvf uClibc-$UCLIBC_VERSION.tar.bz2
ln -s uClibc-$UCLIBC_VERSION uClibc
pushd uClibc

#we have to provide a configuration - it can be done by running make menuconfig just like linux kernel, or you can reuse mine
curl http://julien-rialland.fr/pub/uclibc.config -o .config
make "PREFIX=$TOOLCHAIN_INSTALL_DIR" "CROSS=$TOOLCHAIN_TARGET-" "KERNEL_HEADERS=$TOOLCHAIN_INSTALL_DIR/include"
make "PREFIX=$TOOLCHAIN_INSTALL_DIR" "CROSS=$TOOLCHAIN_TARGET-" "KERNEL_HEADERS=$TOOLCHAIN_INSTALL_DIR/include" install_dev install

#and copy the headers and libraries at the right places into the toolchain's tree : 
cp $TOOLCHAIN_INSTALL_DIR/usr/$TOOLCHAIN_TARGET/usr/lib/* "$TOOLCHAIN_INSTALL_DIR/$TOOLCHAIN_TARGET/lib"
cp -Rf $TOOLCHAIN_INSTALL_DIR/usr/$TOOLCHAIN_TARGET/usr/include/* "$TOOLCHAIN_INSTALL_DIR/include"
popd
popd

Now the build chain is quite complete, we can now build plain executables for our linux system. The only missing thing is the c++ runtime, which is only useful if you are going to run apps written in c++ on your system – as it is not the case with mine, I skipped it. But here are the steps for building it :

C++ runtime

As usual, fetch the sources, configure, and compile !

UCLIBCXX_VERSION=0.2.4
push $TOOLCHAIN_INSTALL_DIR/src
curl "http://cxx.uclibc.org/src/uClibc++-$UCLIBCXX_VERSION.tar.bz2" -o uClibc++-$UCLIBCXX_VERSION.tar.bz2
tar -jxvf uClibc++-$UCLIBCXX_VERSION.tar.bz2
ln -s uClibc++-$UCLIBCXX_VERSION uClibc++
pushd uClibc++
     #make menuconfig, or get my .config at julien-rialland.fr
     make "PREFIX=$TOOLCHAIN_INSTALL_DIR" "CROSS=$TOOLCHAIN_TARGET-" "KERNEL_HEADERS=$TOOLCHAIN_INSTALL_DIR/include"
popd
popd
Catégories :Uncategorized

A minimal linux system – Part 1

28 décembre 2012 Laisser un commentaire

As a challenge to myself, in order to understand the way one can build a linux system, I started to play building a running os from scratch.
The challenge was the following : having a running operating system on a Pc (a virtualbox virtual machine was my target platform, for more convenience),
with a shell, at least a decent set of standard commands (ls, cp, grep, etc…), networking support, and an ssh server.

The main goals were :
– prooving to myself that building a linux Os is easy
– seeing how far one can go in order to have the ligthest linux-based system possible

So I aimed at reducing the size of the system, so I had to cut corners each time it was possible in order to make the smallest binaries. That’s the reason why it does’nt embed fancy hardware detection mechanisms, filesystems support,
software package systems like apt, etc…

x86

I choosed to compile for x86 pc. By changing some options in the following tutorial, it might be easy to setup things in order to compile for another
target platform – arm for instance. Linux kernel configuration and modules compilation is the main thing that make compiling linux for alternative
platforms a bit difficult. The matter is not the processor itself, but the integration between linux and underlying hardware (the arch/ directory in the kernel source tree), which
may be hard to configure properly on exotic boards. Actually i didn’t go into these issues. as x86 pc is a basic platform for linux.
I also choosed x86 because I tought it would produce smaller binaries.

fitted for one machine

Classical kernel and modules compilation that can be found on linux distribution (debian, fedora and so on) are configured to potentially work on a big amount of
different machines, which is not my case. My goal was to be able to run on one (virtual) machine, so the set of supported hardware in my linux image is limited to this
particular machine :
one floppy disk unit
one ide channel with a cdrom drive and one hard disk.
generic intel pro/1000MT network card
no sound support

Disclaimer : You should use buildroot !

Some good project help building cross-compiler based toolchains for various platforms. buildroot (http://buildroot.uclibc.org/) allow to easily generate a complete
embedded linux system. The fact is that I actually wanted to do it myself in order to understand how it works – That’s the reason why I did not use such tool.
If you are just aiming at having an embedded linux system, you should use it instead of following this tutorial, as it is imo the most proper and standard way to do it.
The fact it that it automates a lot of things – As I wanted to customize almost everything, I did not use it – But you shall know that using buildroot is much more efficient !!
Many good tutorials explain how to use it, just google for it !

Let’s begin

Part 1 of this tutorial covers the compilation of the compiler itself : The first thing we need for building executables for another platform is a compiler that can compile code for my target platform. Such activity is named cross-compiling. Actually my development machine is a 64-bits ubuntu-based system,
and I want to compile for ia32 (I could also compile for arm or any instruction set supported by gcc) – So I need a customer gcc + binutils.

GNU binutils

binutils  are programs that allow to manipulate binaries, there is a linker, and an assembler. These will be used by our compiler. compiling it quite easy

apt-get install curl build-essential flex bison

TOOLCHAIN_TARGET=i386-linux-uclibc
TOOLCHAIN_INSTALL_DIR=$HOME/toolchain-${TOOLCHAIN_TARGET}
BINUTILS_VERSION=2.23

mkdir -p $TOOLCHAIN_INSTALL_DIR/build_src
pushd $TOOLCHAIN_INSTALL_DIR/src
curl ftp://sourceware.org/pub/binutils/releases/binutils-${BINUTILS_VERSION}.tar.bz2 -o "binutils-${BINUTILS_VERSION}.tar.bz2"
tar -jxvf binutils-${BINUTILS_VERSION}.tar.bz2
ln -s binutils-${BINUTILS_VERSION} binutils
pushd binutils
./configure "--target=$TOOLCHAIN_TARGET" "--prefix=$TOOLCHAIN_INSTALL_DIR"
make
make install
popd #binutils
popd #toolchain

GCC

The compiler suite itself.
Gcc relies on third party libraries like gmp, mpfr and mpc which have to be retreived separately. The gcc makefile just need the source code of these libs to be
uncompressed over the gcc source code in order to be detected and used during the compilation of gcc itself.

apt-get install zip zlib1g-dev libgmp-dev libmpfr-dev

GCC_VERSION=4.7.0
GMP_VERSION=5.0.5
MPFR_VERSION=3.1.1
MPC_VERSION=0.9

mkdir -p $TOOLCHAIN_INSTALL_DIR/build_src/gcc
pushd $TOOLCHAIN_INSTALL_DIR/build_src/gcc

curl "ftp://ftp.irisa.fr/pub/mirrors/gcc.gnu.org/gcc/releases/gcc-$GCC_VERSION/gcc-$GCC_VERSION.tar.bz2" -o gcc-$GCC_VERSION.tar.bz2
tar -jxvf gcc-$GCC_VERSION.tar.bz2
ln -s gcc-$GCC_VERSION gcc
pushd gcc

curl "ftp://ftp.gmplib.org/pub/gmp-$GMP_VERSION/gmp-$GMP_VERSION.tar.bz2" -o "gmp-$GMP_VERSION.tar.bz2"
tar -jxvf gmp-$GMP_VERSION.tar.bz2
ln -s gmp-$GMP_VERSION gmp

curl "http://www.mpfr.org/mpfr-current/mpfr-$MPFR_VERSION.tar.bz2" -o "mpfr-$MPFR_VERSION.tar.bz2"
tar -jxvf mpfr-$MPFR_VERSION.tar.bz2
ln -s mpfr-$MPFR_VERSION mpfr

curl "http://www.multiprecision.org/mpc/download/mpc-$MPC_VERSION.tar.gz" -o "mpc-$MPC_VERSION.tar.gz"
tar -zxvf mpc-$MPC_VERSION.tar.gz
ln -s mpc-$MPC_VERSION.tar.gz mpc
popd
popd

#gcc configure script should not be run with its source tree as current path ! - we may use a different (empty) directory during compilation
mkdir -p $TOOLCHAIN_INSTALL_DIR/build_obj/gcc
pushd $TOOLCHAIN_INSTALL_DIR/build_obj/gcc

#generate makefile
./configure     --target=$TOOLCHAIN_TARGET \
--prefix=$TOOLCHAIN_INSTALL_DIR \
--with-system-zlib \
--disable-nls \
--disable-shared \
--disable-libssp \
--disable-multilib \
--disable-libgcj \
--disable-libada \
--enable-interwork \
--without-headers \
--with-gnu-ld \
--with-gnu-as \
--disable-decimal-float \
--disable-libmudflap \
--disable-libquadmath \
--disable-libgomp \
--disable-libstdc++-v3 \
--disable-libitm \
--without-ppl \
--without-cloog \
--enable-languages=c,c++ \
--enable-threads=single
make
make install
popd

#and finally add all produced executable to the path.
export PATH=$PATH:$TOOLCHAIN_INSTALL_DIR/bin

Note that at this point, we have do not have support for the c++ language, even if g++ has been created, we bypassed the compilation of the c++ runtime library.
This will be fixed in the next step by relying on a lightweight c++ runtime (ucLibC++) instead of the implementation provided with gcc (named libstdc++-v3).

Catégories :Uncategorized