History of UNIX Project Build Tools

History of UNIX Project Build Tools  

(The following is derived from the HACKING.txt file of the old open source project which I stopped supporting many years ago. R.I.P.)

You might have noticed above that there are SIX STEPS required to do a rebuild after editing configure.in. Why is it so complicated?

最初最初的时候,只有Makefile,并没有configure等工具出现。

You might remember the days when all the dependencies and rules were encapsulated in one file (called Makefile) and no matter what you changed (including the Makefile itself) the make command would figure out how to rebuild everything. That's not true anymore.

Unix版本越来越多,开始要求Makefile具备跨平台的功能:不同平台,对Makefile有不同的要求。这时工具链就必须延伸。

The original Makefile model worked well before the proliferation of many different types of UNIX and the advent of cross-platform compatibility.

To handle all the different types of Unix, the Makefile has to be complex — so complex that it is no longer practical to edit by hand. Furthermore, many things have to be figured out by the machine where the program is going to be built, rather than the machine where the programmer developed it. So, the "meta-rules" for creating Makefiles got too complicated to edit by hand. Eventually there got to be a lot of different types of source files, and a lot of different rules for what to do if you want to change something. The best way to explain this is by going through the history in chronological order — starting prior to the origin of Makefile and make.

为了理解工具链是如何一步步延伸的,就从历史先后顺序看起。

人肉CC搞定

Originally, all changes were made just by changing C source code (in the foo.c and foo.h files) and typing cc to compile it into a binary:

sources -> cc -> binary

用Makefile推导文件依赖,自动编译

Then came compiling and linking as a separate step. This created the situation that if you change just one foo.c file, you only have to re-compile that one file and then link, but if you change a foo.h file you probably had to compile everything. To save time, people figured out how to make a list of rules telling which ".c" files depend on which ".h" files. The make tool was developed, a program that would automatically figure out what needed to be recompiled. make takes a new source file, called Makefile, and also uses all the program source files. Rebuilding still consisted of just one step:

Makefile and sources -> make -> binary

make wasn't quite as smart as it should have been. For example, there's no way to get it to check if Makefile itself was changed. To work around this, programmers started adding a "target" called "clean" (or something similar) that removes all the object files. Then, if you change Makefile (for example, after realizing you left out an "#include" dependency) you can type make clean to force it to remove the objects, and make to recompile everything. Commands like make clean are still used today. (花絮,这里解释了为什么改变了Makefile后应该执行make clean)

多种Unix变种出现,如BSD,AT&T System III,源码和Makefile中需要条件编译/配置(configuration),以适应不同系统。早起做法是把配置集中到一个头文件里面,例如config.h中,或放到Makefile里。

After a few years, different versions of Unix started to exist (like BSD vs. AT&T System III), and people noticed that you had to change the foo.c and foo.h files and Makefile in different ways depending on what type of Unix you were compiling for. Those changes were called "configuration". Manual configuration is tedious — it takes a lot of knowledge and diligence to make all the correct changes for your own particular type of Unix. Eventually it was decided all such changes should be controlled by #ifdef tests (like "|#ifdef BSD_4_1"), and the #defines could be specified in the Makefile| or a header file called config.h (or something similar). Building then required two steps. In the first step, you edit the Makefile to #define each of the defines (like BSD_4_1) that you need on your system:

[plain] view plaincopy
  1.           generic  
  2.         Makefile and  -.  
  3.           config.h      \  
  4. STEP 1.             manually-edit  
  5.                           \  
  6.                            `->  custom Makefile  
  7.                                  and config.h  
  8.   
  9.             Makefile  
  10.                  and   -.  
  11.              sources     \  
  12. STEP 2.                 make  
  13.                           \  
  14.                            `->  binary  


然后大家受不了这么山寨的搞法,发明了专门的配置系统(1992年),它首先探测系统类型,然后生成针对这个系统的一坨宏。这个系统的名字叫configure (Makefile的上一步出现!),它是个shell脚本。


Next, standard "configuration systems" were created. Usually a configuration system was a set of shell scripts that made all the tests and modifications automatically. For example, it is easy to test for BSD version 4.1, and everyone agreed to use BSD_4_1 to indicate you're on that system. So, (in theory) all it takes is one big set of instructions, (typically, a shell script called configure), to test for all the different types of hardware, oeprating systems, libraries, etc. and generate the #defines for that system. The result, for most programmers, was to replace the first manual step with something more automatic.

Because Makefile was now auto-generated by configure, the configure file was what you edited when you wanted to change the Makefile, and the Makefile became an uneditable, automatically-generated file just like the program binary. The two build steps became:


[plain] view plaincopy
  1.  configuration-files  -.  
  2.                         \  
  3. STEP 1.              configure  
  4.                           \  
  5.                            `->  Makefile  
  6.   
  7.             Makefile  
  8.                  and   -.  
  9.              sources     \  
  10. STEP 2.                 make  
  11.                           \  
  12.                            `->  binary  



起初configure要负责两件事:探测操作系统类型;生成Makefile。后来(1994年),这两件事分化为由两个工具来完成:autoconf负责探测操作系统类型;configure负责生成Makefile。autoconf的参数文件叫configure.in;configure的参数文件叫Makefile.in。


Several different types of configuration systems were in place by 1992. Some consisted of a script called configure that did all the tests to see what type of Unix you're running on, then generated the Makefiles. The configure script had to know a lot about the syntax of makefiles, as well as knowing a lot about how to test for different features of operating systems.

Eventually, the job of doing the operating-system tests and the job of creating the Makefiles from "Makefile templates" was split up into two different tools.

By 1994 it was generally agreed that the best tool for the operating-system tests was autoconf. It took one new source file: configure.in and generated a script called configure as output. This configure script, in turn, took one new source file called Makefile.in, and generated Makefile as an output file. At this point the build had three steps that worked like this:


[plain] view plaincopy
  1.           configure.in -.  
  2.                          \  
  3. STEP 0.               autoconf  
  4.                           \  
  5.                            `->  configure  
  6.   
  7. - - - - tarfile is distributed in this form - - - -  
  8.   
  9.           Makefile.in -.  
  10.                         \  
  11. STEP 1.              configure  
  12.                           \  
  13.                            `->  Makefile  
  14.   
  15.             Makefile  
  16.                  and   -.  
  17.              sources     \  
  18. STEP 2.                 make  
  19.                           \  
  20.                            `->  binary  


第一步用autoconf生成configure文件这件事一般只需要做一次,因为系统不会总是变,也不会经常增加对操作系统有新依赖的代码。回想一下,我们下载的很多开源包里面,只需要执行./configure; make; make install就行了,并没有autoconf什么事,就是这个原因。

Note that Step 0 only had to be done if you changed the configuration requirements, like if you added a major new feature that depended on something that is different on different systems (an example would be adding a graphical user interface to a program that was previously text-only). Therefore, the build process was now split into the "user installation" steps (steps 1 and 2) and the "complete rebuild from scratch" (steps 0 1 and 2). Typically, the programmer would perform step 0 and distribute the result to the users, who perform steps 1 and 2. This is indicated above where it says "tarfile is distributed in this form".

到autoconf这一温饱时代后,人们开始奔小康,对方便性提出了更高的要求:configure的输入文件Makefile.in太他妈大了啊!工程下面的所有文件都要写到这里面去,维护起来很费劲(不信你去找个Makefile.in来看看)。于是,大约在1996年,又发明了一个新工具:automake,专门用于生成Makefile.in。automake的输入文件叫Makefile.am。

The weak point in this system was Makefile.in. This had to be a very large and complex file, because it contained all the rules for how to generate a Makefile, and Makefiles were by this point very complex (about as complex as a programming language) and vary a lot from one OS to another. Since Makefile.in was a source file it had to be edited manually. Most of Makefile.in was the same regardless of what program you were building, and programmers found it cumbersome.

The solution to that was automake. It automatically creates Makefile.in from another new source file, called Makefile.am. By 1996, the standard build process had four steps (two for users doing an install and two more for people adding new features) and the steps were:


对于代码维护者,需要涉及到下图中4步;对于开源代码使用者,只需要涉及到后面两步。


[plain] view plaincopy
  1.           configure.in -.  
  2.                          \  
  3. STEP 0-A.             autoconf  
  4.                           \  
  5.                            `->  configure  
  6.   
  7.            Makefile.am -.  
  8.                          \  
  9. STEP 0-B.             automake  
  10.                           \  
  11.                            `->  Makefile.in  
  12.   
  13. - - - - tarfile is distributed in this form - - - -  
  14.   
  15.           Makefile.in -.  
  16.                         \  
  17. STEP 1.              configure  
  18.                           \  
  19.                            `->  Makefile  
  20.   
  21.             Makefile  
  22.                  and   -.  
  23.              sources     \  
  24. STEP 2.                 make  
  25.                           \  
  26.                            `->  binary  


插播一个问题:configure.ac在哪里?答:还没出生呢!

还记得configure.in是怎么来的吗?当年为了给configure减负,把操作系统探测的逻辑独立出来,搞了个autoconf工具。autoconf工具的输入文件叫configure.in。又过了几年,configure.in成了瓶颈,太他妈难写了,不光操作系统,什么库啊,驱动啊,都得管。为了解决这个问题,autoconf扩展了自己的功能,引入了一个叫做aclocal.m4的”宏文件“,这个文件是用一种叫”m4”的语言写的,擅长探测操作系统的方方面面。扩展后的autoconf,组合configure.in和aclocal.m4为输入,生成configure文件。

Over the next couple years, configure.in got bigger and included lots of code to test for lots of different types of libraries, drivers, operating systems, etc. Eventually configure.in became the biggest and hardest-to-maintain file, just like Makefile.in had been. More recent versions of autoconf have solved this by allowing for the use of a "macros" file called aclocal.m4. The "macros" are written in a language called m4, and they contain the rules for performing all sorts of different operating-system tests. As far as the build process is concerned, these can be treated as part of step 0-A, except that you don't ever have to worry about changing the contents of aclocal.m4:


[plain] view plaincopy
  1.           configure.in   
  2.             aclocal.m4   -.  
  3.                            \  
  4. STEP 0-A.               autoconf  
  5.                             \  
  6.                              `->  configure  
  7.   
  8. STEP 0-B.  (automake step, same as above)  
  9.   
  10. - - - - tarfile is distributed in this form - - - -  
  11.   
  12. STEP 1.    (configure step, same as above)  
  13.   
  14. STEP 2.    (make step, same as above)  


aclocal.m4可用aclocal工具直接生成,也可以手写。如果用工具生成的话,就需要引入macros文件作为aclocal工具的输入。

Around the same time it also became common to use a tool called aclocal to generate aclocal.m4, from a directory of macros files called "macros". This added a fifth step to the full build process:

[plain] view plaincopy
  1.           configure.in   
  2.            macros/*.m4   -.  
  3.                            \  
  4. STEP 0-A.            aclocal -I macros  
  5.                              \  
  6.                               `->  aclocal.m4  
  7.   
  8.           configure.in   
  9.             aclocal.m4   -.  
  10.                            \  
  11. STEP 0-B.               autoconf  
  12.                             \  
  13.                              `->  configure  
  14.   
  15. STEP 0-C.  (automake step, same as above)  
  16.   
  17. - - - - tarfile is distributed in this form - - - -  
  18.   
  19. STEP 1.    (configure step, same as above)  
  20.   
  21. STEP 2.    (make step, same as above)  


当新千年的钟声想起,我们终于告一段落,大厦落成。

This was the way things were done by around the year 2000.

Complete list of files and the order in which they are built:


[plain] view plaincopy
  1. ORIGINAL FILES  
  2.    
  3.         the file: configure.in  
  4.  is created from: typed in by hand  
  5.    
  6.         the file: Makefile.am  
  7.  is created from: typed in by hand  
  8.    
  9.         the file: src/adam.c  
  10.  is created from: typed in by hand  
  11.    
  12.         the file: src/adam.h  
  13.  is created from: typed in by hand  
  14.    
  15.         the file: src/anything.c      (any ".c" not listed below)  
  16.  is created from: typed in by hand  
  17.    
  18.         the file: src/anything.h      (any ".h" not listed below)  
  19.  is created from: typed in by hand  
  20.    
  21.    
  22.    
  23.  AUTO_GENERATED FILES  
  24.    
  25.         the file: config.h.in   
  26.  is created from: acconfig.h configure.in acconfig.h  
  27.               by: autoheader  
  28.    
  29.         the file: config.h  
  30.  is created from: config.h.in  
  31.               by: ./configure  
  32.    
  33.         the file: Makefile  
  34.  is created from: Makefile.in  
  35.               by: ./configure  
  36.    
  37.         the file: configure  
  38.  is created from: configure.in aclocal.m4  
  39.               by: autoconf  
  40.    
  41.         the file: aclocal.m4  
  42.  is created from: configure.in macros/*.m4  
  43.               by: aclocal -I macros  
  44.    
  45.         the file: Makefile.in  
  46.  is created from: Makefile.am  
  47.               by: automake  
  48.    

不过,还是那个问题:configure.ac在哪呢?configure.in还得手写?我嘞个去!

表着急,configure.ac只不过是configure.in的新名字而已啦:

At this time aclocal and AM_INIT_AUTOMAKE did not exist, so many things had to be done by hand. For instance, here is what a configure.in (this is the former name of the configure.ac we use today) must contain in order to use Automake 0.20:


原文地址:http://mrob.com/pub/comp/unix-building-history.html


补充阅读:

autoconf的历史:https://www.gnu.org/software/autoconf/manual/autoconf-2.65/html_node/History.html

automake的历史:ttps://www.gnu.org/software/automake/history/automake-history.html#Timeline

整个流程可视化:https://en.wikipedia.org/wiki/Configure_script


特别推荐看看automake的历史,里面有数位作者的回忆,从中可以了解到很多原委。

上一篇:脚把脚教你利用PAI训练出自己的CNN手写识别模型并部署为可用的服务


下一篇:5块钱低成本阿里云大数据生态协同过滤推荐系统实战