This is because Gentoo, by default, requires you to compile all the packages that you want to install. That approach allows you to change compile-time settings. This includes settings for various technologies which would not have been possible in a binary-based distribution.
What is Source Package Management?
Source-based package management is the original way of distributing third-party programs in Linux. This is a system where a maintainer has a list of recipes in obtaining and creating programs. These recipes can contain both a program’s source code and the instructions on how to obtain it. As a user, you can then download these recipes from your distributor through your package manager. Once done, your machine will download the source code of the program and create it in real-time for you. While this might seem old fashioned, doing it this way has a number of benefits:
Distributing only the recipe of a package takes less data than distributing binary copies. This, in turn, reduces the amount of resources required to host a mirror of a repository.Compiling a package from its source code exposes you to the programs that make your operating system work. This can be especially useful if you want to learn how your computer works and how you can extend its functions.Lastly, compiling a package from its source code gives you the ability to customize and change the behavior of your programs. You can easily include or remove features that you do not need.
Despite that, this approach also comes with a number of downsides. For example, packages such as Firefox and Chromium takes hours to compile and install. This is because these are programs with millions of lines of code and hundreds of dependencies. Knowing that, this article aims to introduce five tricks that you can use to improve your compile times when using Gentoo.
1. Optimize MAKEOPTS
One of the easiest ways of speeding up the compile times in Gentoo is by optimizing your make.conf. This is the main configuration file that Portage uses to determine the compilation settings for your machine. As such, it contains a variety of variables that determine your system’s architecture and settings. You can find the make.conf file in “/etc/portag”. You can also print the contents of the file with the following command: The only thing that you need to change to optimize Portage is MAKEOPTS. This is the setting that deals with the optimization flags for your system. Consider the following example: In this case, there are two flags that you can use with MAKEOPTS:
The -j flag indicates the amount of compile jobs that you want to allocate to your CPU. A good rule of thumb when setting this value is to give it half of the amount of cores your CPU has. For example, if you are running a quad core CPU you can set this value to -j2.The -l flag, on the other hand, sets the load average that you want for your system. The load average is the combined amount of tasks that the CPU is doing for an amount of time. For example if you want to speed up the compile times for your system, you can set this value to -l2.
2. Take Advantage of CCache
Another way of speeding up your compilation times is by using CCache. This is a third-party program that allows you to save common compiled files to a temporary space in your computer. From there, your compiler can then use these saved files instead of recompiling them again. This can be especially useful if you are either constantly rebuilding programs or compiling a large package. You can install CCache through Portage: The next thing to do is to setup CCache’s settings. To do that, you need to create the ccache.conf file: From here, you can then use CCache’s default configuration settings to get started: The last thing that you need to do is to tell Portage to use CCache when compiling packages. To do that, you can include the following lines to your make.conf:
3. Try DistCC
Similar to CCache, DistCC is also a third-party program that can help speed up your compilation times. Unlike CCache, DistCC does this by distributing the compile tasks to other computers in your network. This can be especially useful if you have multiple unused computers in your home. The way it works is that DistCC is split into two programs: distccd and distcc.
distccd is the utility that is running in the compute server. This is a simple program that listens for incoming DistCC requests from clients and takes in tasks from them. Once it is done compiling, the compiled object is then sent back to the client to be packaged.distcc, on the other hand, is the client program that connects to distccd. This is the program that works with Portage to seamlessly schedule and distribute compile tasks in a network.
Creating a small DistCC cluster is relatively simple. To do that, you need to install the distcc program in both your server and your client PC:
Configuring a DistCC Server
Next, you need to configure your server to accept DistCC connections. You can do that by creating a distcc configuration file in “/etc/conf.d/”: Once done, you can include the following line of code in your distccd file:
The port flag indicates the port where distccd will listen for incoming connections. 3632 is the default port for DistCC.The log-level and log-file tells DistCC to maintain a log file for the tasks that it is taking.The N flag indicates the “niceness” of distccd. This is a variable that tells DistCC how much it can use the server’s CPU for compile tasks. This is a value that ranges from 0 to 20 where 0 is the least nice and 20 is the most.Lastly, the allow flag tells DistCC to only listen for incoming connections from specific addresses. For example, the line above tells DistCC to only listen for incoming connections from a single client.
The next thing you need to do is to create the distccd log file: With that done, you can now enable your distccd server to accept incoming DistCC connections:
Configuring your DistCC Client
Unlike distccd, configuring your DistCC client is incredibly simple. First you need to tell your client where to look for the distccd server: This will tell DistCC to use both the local machine and the server to compile programs. From there, the only thing left to do is to configure your make.conf to use DistCC. Similar to CCache, you need to use the FEATURES variable to tell Portage that you are running DistCC in your network. For example, this is a make.conf from a client machine:
4. Use BuildPKG
Another way to speed up the compile times in Gentoo is by creating a binary distribution server. In this, you are using a separate machine where packages are built and distributed in binary form. Unlike DistCC, distributing binaries only requires a single machine to do the compiling. The resulting program is then shared across multiple machines where they can install it normally. Doing it this way can be helpful if you are already using Gentoo on a number of similar machines. This is because binaries reduce the time that it takes to build up and maintain a single client. To get started, you only need to add the following line to the make.conf of your build server: This will, in turn, tell Portage to automatically create a binary package for every program that you install in your server. From there, you need to create a way to share those binaries to the machines in your network. To do that, you can install a simple HTTP web server: Next, tell lighttpd to serve your server’s packages by adding the following line to your “/etc/lighttpd/lighttpd.conf”:
Installing Binary Packages in Gentoo
With that done, installing binary packages from your distribution server is a simple two-step process. First, you need to tell Portage the address of your distribution server: From there, I used Portage to install binary packages: Here, the -K option will force Portage to only look for binaries of the package that you want to install. In doing this, it will look at the server if it has any copies of the Firefox program. If Portage cannot find any, the installation will fail. This ultimately means that the server must already have the binary of a program before any clients request for it.
5. Optimize your Compiler in Gentoo
Lastly, you can also set the GNU C Compiler (GCC) options to allow it to optimize for speed. Unlike MAKEOPTS, however, these options will depend on the machine that you have. This includes the CPU as well as the hard disk and memory that you have. Further, you need to place these options in the COMMON_FLAGS variable. For example: In this, there are three options that you can set to optimize GCC:
The march option tells GCC to compile the program in the CPU that you want. In most cases, you tell GCC to create the program for your local CPU. To do that, you can give the value “native” and GCC will build for the CPU that it is currently running on.The O option tells the level of optimization that you want for your programs. This is a range between 0 to 3 and the higher the value that you set the faster the program will run. However, doing that will increase the time it takes to compile it.The pipe option is a toggle that you can set to speed up the compilation process. It does this by holding objects in RAM instead of saving it in temporary files. As such, this will consume a lot of memory and using this with limited RAM will crash GCC.
1. I do not have a MAKEOPTS or FEATURES variable in my make.conf. Is my Gentoo install broken?
No! It is normal for the default make.conf file to not have the MAKEOPTS and FEATURES variables. You can just add the variables and its values to your make.conf and Portage will recognize it and apply it immediately.
2. Is it possible to only enable CCache for the packages that I want?
Yes! You easily enable CCache only for specific packages. To do that, you need to first remove the “ccache” value in your FEATURES variable. From there, you need to edit the “/etc/portage/package.env” file. This is the file where you can enable Portage features on a per-package basis. Knowing that, this is a package.env file that uses CCache for some programs: In doing so, I enabled CCache for Firefox, GCC and Clang. This means that Portage will only save the object files for these programs and their associated dependencies.
3. Are there any issues with running a Gentoo distribution server?
One of the main issues of maintaining a distribution server is that you are responsible for keeping the binary packages up to date. This means that before you can update any clients you will need to update the distribution server first and then remove the old binaries in your clients. Another issue with running a distribution server is that you lose the flexibility of USE flags for your client machines. This means that you need to enable flags that you do not need on all of your machines. Image credit: Unsplash
