The Sacred & the Profane: the search for simplicity in the total hardware-software combination

If you haven’t done so already, you may want to start by reading the Preface to Knowledge Engineering & Emerging Technologies.

---

January 31st, 2024 (4th ed)¹

The aim of this article is to encourage you to take an end-to-end perspective in your designs, seeking to minimize the overall complexity of your system, of the hardware-software-user combination. To achieve this, it is helpful to understand how computing, and within that, how the notions of the sacred and the profane have evolved over the past 60 or so years.²

The following remarks set out a ‘true north’ perspective for this conversation:

1. “We are reaching the stage of development where each new generation of participants is unaware both of their overall technological ancestry and the history of the development of their speciality, and have no past to build upon.” – J.A.N. Lee, [Lee, 1996, p.54].
2. “Any [one] can make things bigger, more complex. It takes a touch of genius, and a lot of courage, to move in the opposite direction.” – Ernst F. Schumacher, 1973, from “Small is Beautiful: A Study of Economics As If People Mattered”.³
3. “The goal [is] simple: to minimize the complexity of the hardware-software combination. [Apart from] some lip service perhaps, no-one is trying to minimize the complexity of anything and that is of great concern to me.” – Chuck Moore, [Moore, 1999] (For a succinct introduction to Chuck Moore’s minimalism, see Less is Moore by Sam Gentle, [Gentle, 2015]
4. “The arc of change is long, but it bends towards simplicity”, paraphrasing Martin Luther King.⁴

The discussion requires a familiarity with lower-level computing, i.e. computing that is close to the underlying hardware. If you already have some familiarity with this, you can jump straight in to section 2. For all backgrounds, the discussions in the Interlude (section 4) make for especially enlightening reading. Whether you find yourself in violent agreement or disagreement, your perspective is welcomed in the comments!

1. Why knowing (some) low-level software development is still important in 2024.

You might find it surprising to hear that technologists, students, and managers should understand low-level software development at the end of the first quarter of the 21st century.

It’s not about coding a large system in assembly language, or manually disassembling machine code (though you should know enough to know that you could if you have to). It is about ensuring that you have the historical and cultural grounding to be able to distinguish between good and poor decisions, that you can understand the mistakes of the past and discern when the current thinking is trending towards re-making the same mistakes in a modern context. It’s about being able to make the right value judgements when thinking about design choices so that you can have a wise influence on the technical directions the products you work on should take. For programmers, it is about ensuring that you have ability to do good computing work without needing to be supported by the $1,000/yr IDE annual subscriptions and the 100,000 lines-of-code libraries. It’s about knowing that you can rely on your own skills and that you have the ability to talk to a machine directly (bare metal), without requiring 10-15 layers that typically sit between the average programmer and the machine.

Despite the advances in language design and programming automation over the past 50 years, acquiring familiarity with some low level languages is essential for the rounded education of a technologist, and is an essential part of technical literacy. Indeed, one might argue that many more citizens in our society, and certainly all students, should have formative experiences in computing⁵ at a level close to the machine, something which is needed to shift the perspective from being merely consumers of technology to being creators themselves.⁶

So,why invest in understanding low-level software engineering?

Six reasons:

1. familiarity with at least one an assembly language exposes the student to the machine itself, which has been historically, and continues to be, the foundation of computing,
2. assembling human readable instructions to create machine code is the essential purpose of all software development toolchains (pre-processor, compiler, assembler, linker),
3. assembly languages are the least common denominator for embedded systems programming and they contain the instructions that a processor (ALU or CPU) understands,
4. assembly language instructions are the interface between electrical engineers building digital logic subsystems of a chip and programmers,
5. assembly languages and the chips they communicate with are the cornerstone of the bootstrapping mindset of Forth in the pursuit of a minimal computing system, minimal hardware, and minimal software, upon which all else can be built. If you wanted to program bare metal (without an operating system or other software layer in the way), you would need to drop into assembly. (One way that works is to roll your own minimal Forth, even a minimal ‘3-instruction Forth’ tethered to your native computer via a serial connection); and finally
6. assembly language provides a common denominator for the mathematical analysis of algorithms (cf. Knuth’s The Art of Computer Programming).

From assembly language, two directions will open themselves up for you much more easily than otherwise as you will have the cultural and experiential grounding of computing at a fundamental level. The first direction is the road upward into programming and computer science: assemblers, compilers, parsers, and higher level languages; the other direction is taking the road downward into microcontrollers, digital logic, sensors, and embedded development (see Appendix 1).

How to go about gaining low-level experience?
Viewed in the right way and armed with the right toolchain, becoming comfortable with an assembly language is not that difficult. If you have grit and perseverance (and time!), I suggest to go on a couple of months self-study foray into an assembly language (x86 makes sense if you’re on a Windows platform, Atmel 328 AVR if you’re partial to getting into Arduino and embedded systems), side-by-side with a special high level language like Forth, which only wraps lightly around the machine itself (see the Related Readings section). Remember that you don’t have to try to become a veteren assembly language programmer. The important thing is to let the experience expand your mind.

Why Forth? Forth is an exceptionally elegant language for programming in, whether for embedded systems, or any other system type application. Forth can easily drop into assembly language, while retaining the efficiency and elegant design of a high level language or domain specific language (DSL). Forth is lightweight, can be used for both low- and high-level programming, and provides a re-usable investment that will not become obsolete (Forths have been written for most popular controllers and processors, and if one does not exist, it is not too difficult to write a stripped down 3-instruction Forth, e.g. for Arduino Atmega328P). Furthermore, writing a frugal, minimalistic Forth virtual machine for a bare metal chip is a fantastic way to build the real capability that assembly language programming gives you: the confidence that you can take ANY chip, its documentation, and bootstrap your way to a higher level language for that chip, all without requiring an operating system or any other facilities. Just you and your serial port and your knowledge of assembly language and Forth. This is the real computing power that can be in your hands. Freedom from depending on hundreds of complex elements and libraries and billion dollar chip foundries. A $10 Arduino chip, a serial port, and some documentation.

That’s sufficient context for now. Let’s get back to the storyline.

2. The Sacred and the Profane: How the approach to software engineering has shifted

The rise of the engineering mindset: structure, reliability, and a systems perspective are sacred
The programmers and computer scientists of the 1950s were acutely aware of the human inefficiencies of programming in an assembly language and its non-portability, and in the 1960s and 1970s developed ground-breaking approaches to computing to reduce this situation: (1) structured programming principles (Niklaus Wirth, Edsger Dijkstra, Tony Hoare) that allowed programs to be written in a way that resembled a form of logical specification of behaviours, which then could be checked formally for correctness, independent of the computer chip that was being targeted; (2) languages like C which aimed to be universal assemblers in which one could write large and complex multi-user operating systems (MULTICS, UNIX) and compile down to a large number of target chips; (3) languages like Forth which distilled computing down to essentials of definitions (words) that passed essential parameters on a virtual stack machine, leading to highly factored (simplified) programs that could be run on any chip with a minimum of chip-specific support code written in assembly; (4) the development of RISC (reduced instruction set) and MISC (minimum instruction set) computers to combat the rise in complexity of fundamental operations; (5) the development of LISP which, like Forth, re-imagined computing to essentials of lists and recursion, leading to concise formulations of algorithms such as Quicksort and AI routines.

All of these developments sought to improve the ability of human beings to approach problems involving computing while remaining insulated from the details of the specific chip and its specific instruction set and the specific timings involved. Of course such details could not be completely banished, they could be dealt with once, building upwards from the chip itself and presenting to the programmer a chip-independent interface which the universal computing approach would then target. The question then was ‘what should be this chip-independent, universal computing, logical, pragmatic interface and what language should such commands we written in?’ This kicked off an evolutionary golden age of higher level programming languages that all sought to explore a particular perspective on the problem.

What unites all of these higher level languages is that source code written in them has to be translated through pre-processors and compilers into assembly code, and thence be assembled and linked to result in executable machine code that can run on the particular chip.

3. From the sacred to the profane: a cultural shift from engineering to theory: the rise of complexity and compiler optimization

A shift happened in the 1990s. Processors started to become much more complex, backed by large marketing budgets (e.g. Intel), targeting large corporations with even larger IT budgets (e.g. companies which hired IBM as consultants). This trend was mirrored with operating system software that was also rapidly becoming larger and more complex and targeting those same large IT budgets (e.g. Microsoft in the switch from DOS to Windows).

This shift altered the perspective of the software toolchain developers. Prior to this, and up to v2.8 (March 1998), the GNU C Compiler (GCC) had as its mission to produce free, open-source, high quality, portable, universal assemblers in the engineering perspective. A programmer should be able to use the C language and the GCC toolchain to produce clean assembly code for multiple machines, inspect it, and see the correlation between the higher level code blocks that had been written, and the assembly code it compiled down to, and from there, the machine code it assembled to.

In 1997, the idea of an optimizing compiler and a compiler able to handle a broad array of languages began to attract significant energy but which also eroded the FSF’s commitment to one-to-one mapping between high-level code blocks and assembly code. With the Experimental branch of GCC that ultimately became 3.0, it was no longer possible to know exactly how portions of C code would be translated into assembly. The claim was that these object codes were faster code (even if not smaller) by exploiting machine specific tricks during compilation. This trend allowed processor designers to design ever more arcane features and provide special optimizing compilers to exploit those features as part of a sales strategy designed to capture and retain market share.

Relaxing the correspondence between high-level input code and low-level output code introduced problems of verifiability. Adding templatized sections of assembly code, thousands of lines long, with entry gates guiding which instruction path would be followed, bloated code size. Depending on the optimizing compiler switches that were set, the same high-level code could compile to very different assembly code. With each new major compiler version, regression tests (same code, same settings) on assembly code would break, leave alone regression tests on machine code.

4. Interlude

Before jumping ahead, if you feel you need more history, real-life familiarity with what’s happened between the 1980s/1990s and today, then you may want to read each of these where the storyline forks and you can build up that background before proceeding. Below are 6 masterpieces of ‘history traversing, time-lapse-photographs-in-text-form’:

(1) Software Disenchantment [Tonsky, 2018]

(2) The Story of the IDE and Editor: The IDEs from 30 years ago, compared to today. [Merino, 2023]

(3) The Story of Programming in the 21st Century: Programming Modern Systems Like it was 1984 [Hague, 2014], and The Duct-Tape Programmer (Worse is Better, Shipping is a Feature) [Spolsky, 2009]

(4) The Injustice of Complexity in Computing [Nicholi, 2020], or his follow-up article, An Ethos for Sustainable Computing, [Nicholi, 2020]

(5) Those Plain Old Webpages, [Plesioanu, 2017]

(6) Small Software is Beautiful, [Hoyt, 2021]

And perhaps the most thought-provoking of them all:

(7) The Waning Days? A Eulogy to the Mundane as Programming enters the age of Chat-GPT4 , by Jamie Somers, Nov 2023, New Yorker

(There’s a few more in the endnote.)⁷

If you feel you’ve got the pulse, or would rather press onward, you can read Rob Pike’s 5 Rules of Programming reprinted below (yes, just five).

Rob Pike’s 5 Rules of Programming
Rule 1. You can’t tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don’t try to second guess and put in a speed hack until you’ve proven that’s where the bottleneck is.
Rule 2. Measure. Don’t tune for speed until you’ve measured, and even then don’t unless one part of the code overwhelms the rest.
Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don’t get fancy. (Even if n does get big, use Rule 2 first.)
Rule 4. Fancy algorithms are buggier than simple ones, and they’re much harder to implement. Use simple algorithms as well as simple data structures.
Rule 5. Data dominates. If you’ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.
Pike’s rules 1 and 2 restate Tony Hoare’s famous maxim “Premature optimization is the root of all evil.”
Ken Thompson rephrased Pike’s rules 3 and 4 as “When in doubt, use brute force.”.
Rules 3 and 4 are instances of the design philosophy KISS (Keep It Simple and Small).
Rule 5 was previously stated by Fred Brooks in The Mythical Man-Month. Rule 5 is often shortened to “write stupid code that uses smart objects”.

Ok, on we go!

5. Disillusionment in the face of culture change: quotes from prominent computer scientists

For those computer scientists who had been there from ‘the very beginnings’ of the field of computing and worked hard to build a precise, rigorous, structured discipline of software engineering from the more free-wheeling days of assembly language code, this shift again to looseness on reliability and cavalier disregard for the size and speed of software, was highly dispiriting. The remarks below express the mood. Commentary is in the endnotes.

1. Tony Hoare said in 1974:
   “So it is among the prophets of doom that I wish to enrol myself. I believe that the current situation in software design is bad, and that it is getting worse. I do not expect that this will be recognised by the designers, manufacturers, sellers, or even the users of software, who will regard the increase in complexity as a sign of progress, or at least an inevitable concomitant thereof; and may even welcome it as a tribute to their intelligence, or at least a challenge. The larger manufacturers are not only set in their ways, but they have actually profited from the increase in complexity of their software, and the resulting decrease in the efficiency and effectiveness of their customers’ use of hardware. And everything is now so complicated, that any particular attack on the problem of low quality software design can always be evaded by appeals to tradition, to standards, to customer prejudice, to compensating advantages, and even to promises to mend the fault in future issues.
   So it will be a long time before there is even any recognition of the problem which faces our profession. But even if the problem were widely recognized and deplored, its solution is going to present extremely difficult technical problems. The pursuit of complexity is easy, and the implementation of complexity can safely be delegated to competent managers. But the pursuit of simplicity is one of the most difficult and challenging activities of the human mind. Progress is likely to be extremely slow, where each complexity eliminated must be hailed as a breakthrough. we need not only brilliance of intellect but breadth of experience, nicety of judgement, excellence of taste, and even more than our fair share of good luck. And finally we need a puritanical rejection of the temptations of features and facilities, and a passionate devotion to the principles of purity, simplicity and elegance.”⁸

2. “We are building a culture which can not survive as trivial an incident as Y2K. Once we lose the billion dollar fabrication plants and once we lose the thousand man programming teams how do we rebuild them? Would we bother to rebuild them? Are computers worth enough to demand the social investment that we put into them. It could be a lot simpler. If it were a lot simpler I would have a lot more confidence that the technology would endure into the indefinite future.” – Chuck Moore, 1999, 1x Forth ⁹

3. “There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.” – C.A.R. (Tony) Hoare, The Emperor’s Old Clothes, p.9, Communications of the ACM, 1981

4. “A most important, but also most elusive, aspect of any tool is its influence on the habits of those who train themselves in its use. If the tool is a programming language this influence is, whether we like it or not, an influence on our thinking habits.”
   – Edsger Dijkstra ¹⁰

5. “The designer of a new system must not only be the implementor and the first large-scale user; the designer should also write the first user manual. … If I had not participated fully in all these activities, literally hundreds of improvements would never have been made, because I would never have thought of them or perceived why they were important.” – Donald Knuth ¹¹

6. “Good code is its own best documentation. As you’re about to add a comment, ask yourself, ‘How can I improve the code so that this comment isn’t needed?’ Improve the code and then document it to make it even clearer.” – Steve McConnell Code Complete

7. “Programmers are always surrounded by complexity; we cannot avoid it. Our applications are complex because we are ambitious to use our computers in ever more sophisticated ways. Programming is complex because of the large number of conflicting objectives for each of our programming projects. If our basic tool, the language in which we design and code our programs, is also complicated, the language itself becomes part of the problem rather than part of its solution.” – C.A.R. (Tony) Hoare, The Emperor’s Old Clothes, p.10, Communications of the ACM, 1981

8. “[On the software side] My contention is that every application that I have seen that I didn’t code has ten times as much code in it as it needs. How big should a program be? About a thousand instructions seems about right to me to do about anything.8 How do you get there? What is the magic? How can you make applications small? (1) No Hooks. (2) Don’t Complexify. (3) Factor. You factor. You factor, you factor, you factor and you throw away everything that isn’t being used, that isn’t justified. The whole point of Forth was that you didn’t write programs in Forth you wrote vocabularies in Forth. When you devised an application you wrote a hundred words or so that discussed the application and you used those hundred words to write a one line definition to solve the application. It is not easy to find those hundred words, but they exist, they always exist.”- From a 1999 talk by Chuck Moore to the Silicon Valley FIG, titled “1x Forth”

9. “Our unconscious association of elegance with luxury may be one of the origins of the not-unusual tacit assumption that it costs to be elegant. To show that it also pays to be elegant is one of my prime purposes.” – Edsger Dijsktra, EWD-0117, Programming Considered as a Human Activity

10. Wirth’s Law: “Software is getting slower more rapidly than hardware becomes faster.” Niklaus Wirth (author of Pascal, Oberon, etc.), rephrasing of Martin Reiser’s “The hope is that the progress in hardware will cure all software ills. However, a critical observer may observe that software manages to outgrow hardware in size and sluggishness.” ¹²

11. “The goal [is] simple: to minimize the complexity of the hardware-software combination. As far as I can see no-one else is doing that. Some lip service perhaps, but no-one is trying to minimize the complexity of anything and that is a great concern to me.” – Chuck Moore, 1999, 1x Forth ¹³

6. The Story of the GNU C-Compiler: What had been sacred before (human verifiability, certainty) has now become profane

How C changed from a Portable Universal Assembler to an Automatic Code Optimization Engine
The desire to more easily port the UNIX operating system to different machines led to the creation of C as a programming language that could remove the need for assembly coding in most situations. C was designed during the transition to more structured programming. The early C compilers were viewed as portable or universal assemblers. This view held to 1999 when the EGCS group began working on GCC 3.0, embracing compiler optimization techniques and automatic code improvement as a primary goal as opposed to universal assembly.

How we got to where we are: a Short History of Assemblers, C, Linux, and GNU C, and GCC

1. First assembler (Initial Orders) written for the EDSAC computer by David Wheeler (1948). First assembly language concept developed by Kathleen Booth (1947) at Institute for Advanced Study, Princeton (Booth-1947.pdf).
2. Fortran is developed, with the first commercially available compiler (1957, John Backus, IBM)
3. The C language was invented by Dennis Ritchie at AT&T Bell Labs (1969-1973) to accelerate the writing of the UNIX operating system (Ken Thompson) for new computers and adopted the notion of a computer language independent of the particular computer. The first C compiler was written in assembler, and then bootstrapped. UNIX was rewritten in C in 1973. Brian Kernighan co-authored with Ritchie the first C standard (1978), followed by the ANSI C standard (1983).
4. Andrew Tanenbaum at the Free University of Amsterdam developed the Amsterdam Compiler Kit (ACK) in 1983, one of the first portable and retargetable compilers, i.e. able to be compiled onto many machines and to accept source code written in many different languages.
5. Richard Stallman was inspired by Tanenbaum’s Amsterdam Compiler Kit and asked to GNU it, but was turned down. This set him on a path to developing the GNU C Compiler for his GNU Unix (1987). | Landscape of C compilers before 1987 GNU C | GCC Early History
6. Andrew Tanenbaum developed the MINIX operating system (Mini-UNIX) (1986). Read his 30 years later retrospective lessons learned from MINIX (2016). Over time, MINIX led to further OS development and a very nice OS (TinyOS?) and real-time OS. The modern MINIX is based on the notion of a microkernel — something so small that it is easy to understand and much harder to fail, having only 5000 lines of code. This is MINIX 3 — available for free. Minix 3 is comprehensively redesigned to be “usable as a serious system on resource-limited and embedded computers and for applications requiring high reliability.”
7. Linus Torvalds as a student studied MINIX closely, and was inspired by this to develop Linux, which he offered free to anyone using MINIX (August 1991). How Linux was born, told by Linus himself | Key difference: monolithic kernel (Linux), microkernel (Minix)

A simplistic view of how GCC works

1. parse source code
2. decide what language it is and submit to that language’s front-end, a separate program.
3. “Each front end uses a parser to produce an [abstract] syntax tree of a given source file.”
4. “Currently all front ends use hand-written recursive-descent parsers” to produce a tree that is fully independent of the processor being targeted.
5. These are, if necessary, converted to the middle end’s input representation, called GENERIC form; the middle end then gradually transforms the program towards its final form.
   The back-end then takes the optimised generic form and applies a pattern matching algorithm to it to produce machine code.

Timeline showing the shift in perspective in GCC versions

GCC – Original Perspective:

1. 0.9 Mar 22, 1987
2. In May 23, 1987, the first version of GCC was released (1.0) — and Minix was released
3. By December 1988, was on 1.32 after the initial flurry of development, and development slowed from a release once every month prior to this, to every 2-9 months afterwards until 1.4.2
   Richard Stallman author
   Christopher Fraser for the idea of using RTL as an intermediate language
   Paul Rubin (of clf fame?) for writing most of the preprocessor

4. By 1991, GCC 1.x had reached a point of stability but architectural limitations prevented many desired improvements, so the FSF started work on GCC 2.x
5. 1.42 is the last version that was C only. (20.Sep.1992)
6. In 1992, GCC version 2 was released which supported C, C++, Obj-C. Prior to this the C++ compiler was released separately from the C compiler
7. Feb 22, 1992
   The FSF kept such close control on what was added to the official version of GCC 2.x that GCC was used as one example of the “cathedral” development model in Eric S. Raymond’s essay The Cathedral and the Bazaar.

8. In 1997, a group of developers formed EGCS—Experimental/Enhanced GNU Compiler System[15][16]—to merge several experimental forks into a single project. The basis of the merger was a GCC development snapshot taken between the 2.7 and 2.81 releases (2.7.2.3). Projects merged included g77 (Fortran), PGCC (P5 Pentium-optimized GCC), many C++ improvements, and many new architectures and operating system variants.[17][18]
9. 2.7.2.3 Aug 22, 1997
   Marc Lehmann
   Mission statement:
   Further Development of GNU Compilers
   New languages
   New optimizations
   New targets
   Improved runtime libraries
   Various infrastructure improvements

10. EGCS development proved considerably more vigorous than GCC development, so much so that the FSF officially halted development on their GCC 2.x compiler, “blessed” EGCS as the official version of GCC and appointed the EGCS project as the GCC maintainers in April 1999. With the release of GCC 2.95 in July 1999 the two projects were once again united.
11. GCC has since been maintained by a varied group of programmers from around the world under the direction of a steering committee.[19] It has been ported to more kinds of processors and operating systems than any other compiler.
12. 2.81, (2 Mar 1998), was the last non-EGCS release, but included C++ support

GCC – New Perspective:

1. In 2001, GCC version 3 was released incorporating ECGS (Experimental GNU Compiler System), with improved compiler & parser optimizations.
2. After 2.81 it was unified with EGCS, starting with 2.95 and support for MANY languages — Fortran, Objective-C (Apple), even Java. C, C++, Obj-C, CHILL, Fortran (g77) and Java (gcj)
3. 2.95 July 31, 1999
4. All manner of additions were brought in in v3.x
5. 3.4.0 April 2004 was the first version to require a C89 compiler to build
6. Last v3 version was: v3.4.6 (Mar 2006)
7. In 2005, GCC version 4 was released. As of July 2012, the latest release of GCC is 4.7.4.
   The intermediate tree representations were simplied with the introduction of GENERIC and GIMPLE, two new forms of language-independent trees that were introduced with the advent of GCC 4.0.

8. In GCC 4.x, the syntax tree produced by each language front-end is independent of the machine, so is a true universal machine independent language, i.e. true portable intermediate language which can then be compiled down to machine code efficiently. So GCC has gone the direction of computer science, identifying a truly machine independent language representation to which all (or mostly all) languages can be automatically reduced, and which itself can then be automatically compiled down to machine code to create efficient machine code… This is a much broader objective than before.
9. Today the languages supported include:
   C, C++, Objective-C, Objective-C++, Fortran, Java, Ada, Go, Pascal, Mercury, Modula-2, Modula-3, PL/I, D, and VHDL.
   But GCC is now quite a tower of babel, using Perl, Flex, and Bison in its own toolchain, and incorporates GMP, MPC, and MPFR.

10. In May 2010, the GCC steering committee decided to allow use of a C++ compiler to compile GCC.[24] The compiler was intended to be written in C plus a subset of features from C++. In particular, this was decided so that GCC’s developers could use the destructors and generics features of C++.[25]
11. In Mar 2011, 4.6.0, support was added for the GO language
12. In August 2012, the GCC steering committee announced that GCC now uses C++ as its implementation language.[26] This means that to build GCC from sources, a C++ compiler is required that understands C++ 2003.
13. As of Mar 2013, 4.8.0, was first version to require a C++ 98 compiler to build
14. As of 2015 April, GCC is at 5.1, with experimental JIT compilation added.

How the vision for C changed as GCC changed

The following extracts from a discussion on comp.lang.forth (9.Apr.2014) capture the tension in the two different views of C and GCC

Anton Ertl (Technical University of Wien):
“C was a portable assembly language for a long time after 1977. We selected GNU C for Gforth in 1992 because it was a portable assembly language then, and Linus Torvalds selected GNU C for Linux in 1991 because of that. I am pretty sure that that’s also what Richard Stallman intended. The attitude change came later (maybe with egcs, maybe around gcc-3.x), and the effects of that change are now trickling piecemeal into the compiler, and causing one problem after another.”

Andrew Haley (RedHat):
“C was, I suppose, more or less infix PDP-11 assembly language until C90. At that point C acquired an abstract definition and the simple mapping from C to instructions was gone. Simple C compilers which maintained that pre-C90 mapping continued to exist for a while, but over time it has gone.”

Todd Hughes:
“It’s remarkable that most posters in c.l.f seem to understand what most posters in c.l.c don’t: C is more of a worldview and possibly a VM than it is a language. It is no longer helpful to understand the hardware if you want to use C. Actually, it’s probably harmful. C has its own abstract view of the hardware and to use C you have to share that view, and of course be aware of all the broken beer bottles, banana peels and oil slicks left on the road by C. Especially gcc. Forget whatever else you know and put aside any expectations you may have. You will be assimilated.”

My view

1. Optimizing compilers add an additional layer, the optimizer, which a programmer now is implicitly coding to.
2. Optimising compilers blur the mapping between instructions and the machine, claiming to know better than the programmer. The mantra is that the optimizer is transparently making things better. And for less capable programmers (who arguably should have to keep learning), this may indeed be true.
3. But because optimizing compilers tune code automatically, what they DO do is introduce bugs, subtle ones, ones that you would have little hope of catching because you didn’t even know it was being done. For capable programmers, optimizing compilers introduce pernicuous effects stemming from the law of leaky abstractions (see Joel Spolsky’s famous article).
4. Before, one used a portable assembler so that one would not have to learn the intricacies of every machine. One could just learn a portable language and trust that the compiler would map down to assembly language in a straight-forward, repeatable fashion. But this pragmatic engineering perspective was perhaps a bit too boring for the theoretical computer scientists who were eager to progress their own life interest in how to make automated compilation super smart, folding in optimizations that hardware providers with large marketing budgets were happy to encourage as it gave them a “feature advantage” over their rivals, non-essential, but nonetheless something to differentiate themselves with to win highly lucrative big corporate IT contracts.
5. What started out as a portable assembler, has now been evolved to be a language agnostic automatic programmer… We’re now up in the heights of computer science. It is the hubris of “one compiler to rule them all”. GCC onwards from 3.0 is a VERY complicated tower of software.
6. What drives this complexity? One reason is that context and intent are difficult to model. An optimising compiler makes a number of assumptions, which necessarily limit the types of contexts / intents it understands and/or respects. A second problem is that if an optimising compiler were to work on exact equivalence in all possible contexts, not only would this be harder, but there would probably be many fewer optimisations it could legitimately make.
7. By relaxing the requirement of exact equivalence, the best one can say is that modern optimising compilers appear to be working on a probabilistic notion of correctness: “the following is probably right in most cases.” This is a fundamental shift in the notion of reliability that used to underpin software engineering in the early days of mathematically competent computer programmers working on engineering problems (Dijkstra, Hoare, Wirth, Knuth, Moore).
8. Many of the optimisations are not actually improvements in *all* cases, would also be considered “acceptable” in this new probabilistic notion of quality. “better” is not better *always* but better “in many cases along at least one measure”, speed, size, … But going down that road seems like a source of endless difficulties since by definition edge cases will “probably” never be right. This raises questions about whether reliability will ever be achievable when using an optimising compiler. What have we gained and what have we lost in exchange? Dijkstra would “probably” be turning over in his grave.
9. Dijkstra viewed optimizations, on the one hand, and reliability and trustworthiness, on the other hand, as opposing goals. He described optimizing as “taking advantage of a special situation” (cf. specialization). And, according to him, the construction of an optimizing compiler is “nasty”, in comparison to “straightforward but reliable and trustworthy” compiler technology [71, p.538].
10. The real problem is the law of leaky abstractions. In the past you had to know the machine to know how to write good code. Now you have to know the machine *and* the compiler — you have to know enough to ensure that the compiler doesn’t misinterpret what you mean, and you have to now code defensively against the compiler — which is always changing.
11. Given the choice above, knowing the machine and using a language with a toolset that does not take liberties to make probabilistic improvements, is, for me, the better choice. It is a fixed target and so long as the chip maker respects backwards compatibility you are safe. When everything is software, which is easy to change, everyone has an opinion, and there is no longer a fixed target.
12. Can we roll back to the way that C was? Unlikely. The GCC community has almost fully changed.
13. What can be done? Use a smaller compiler, like TCC. Fork GCC before 3.0 and develop that. Or look at a different language. (The GO language group out of Google seems to have gone through exactly this sort of thinking in its early stages. http://golang.org/ | http://blog.golang.org/4years | Go: 2009 – first release | Why Program in Go?)

---

7. Takeaways: The ABCs of ‘bad’ engineering: Three Pitfalls to Avoid

What should be taken away from the discussion? There is no simple prescription, but hopefully you have started to see some themes in what to avoid: (1) indulging in excessive Abstraction, (2) permitting code and feature Bloat, (3) failing to challenge unnecessary Complexity. Abstraction, Bloat, Complexity. I call these the ABC’s of bad engineering. Let’s look at each in turn.

1. Abstraction: Indulging in excessive Abstraction. The pitfall is the intellectual attraction of abstraction. It simplifies concepts, hides information. You can buy into “I don’t need to know.” But as a programmer you should also want to know what is behind that abstraction. What is required to support it. You see, a program cannot escape the fact that it runs on hardware, which means fundamentally it talks to chips which require you to speak their language. So every abstraction away from this is meant to make it simpler for a human to tell a computer what to do. This is good, and indeed, as it should be. But every abstraction requires maintenance to preserve the illusion that you will not need to know what that under the hood. That illusion breaks far too often when dealing with a real problem, and you find yourself having to go beneath the hood anyway, and that’s when you see the hundreds or thousands of lines of code, snaking through calls to one library and another, and you start to face the possible horror. Joel Spolsky has a superb explanation, that he calls “The Law of Leaky Abstractions“. Tony Hoare’s manager, diagnosing his first catastrophic failure as a Chief Engineer, put it very bluntly: ‘”You know what went wrong?” he shouted (he alwyas shouted). “You let your programmers do things which you yourself do not understand.” I stared in astonishment. How could one person ever understand the whole of a modern software product? I realized later that he was absolutely right; he had diagnosed the true cause of the problem and he had planted the seed of its later solution.’ – [Hoare, 1981, p.6]
2. Bloat: Permitting code and feature Bloat – The second problem I see is one arising from a library mentality, or we could call it the wrapper problem. Individual programmers try to write the last word in functionality associated with a particular problem or topic. It has every possible feature one might require. Who is its customer? An abstract “client programmer” who wants to do something with X. The purpose of the library is to ensure that the client programmer never has to know anything more about the implementation. He just needs the documentation that tells him what routines are available, how to call them, and what they do/return. That’s it. Done right, this means client programmer goes away happy, and library writer feels proud of the wrapper he has built. The issue arises when you have wrappers wrapping other wrappers. In principle, wrappers are the right thing. When you are wrapping something complicated, it makes sense to write it once, carefully, robustly, and then provide the wrapper for others to use. This is the principle of information hiding, of abstracting the interface from the implementation. But the problem of bloat arises when the library is a kitchen sink for all possible routines. Rather than pull out only what’s needed, the temptation is to chuck it all in. So you get hundreds of DLLs being included, hundreds of thousands of lines of code, because the wrapper is contained in a container that has a lot more of what you don’t need. So again, the unintended consequence of the right idea is bloat and slowness of software. How many of you have had to write GUI code that produces native GUIs for Windows? How often have you had to abandon the documentation for whatever wrapper system you are using and drop down into the Win32 call library, and work your way through the C-code and descriptions? It has happened for me more often than I care to remember, and is why I approach wrappers and libraries with great caution. You can spend your time learning someone else’s library, or you can learn the fundamentals and do what it is you need (you rarely need more than 20% of all that is in a library – making 20% up as an indicative number).
3. Complexity: Failing to challenge unnecessary Complexity – This third pitfall is aggravated by the earlier two. The more you write libraries, the more you need to document, and test. The more you throw in large frameworks of code, the more likely you are to find an arcane bug or defect. You need to write test harnesses that are equally abstract. When you code abstractly, for example, you code up objects, then each object is now self-contained, which means you need to supply it with a lot of methods, it needs to have its own error checking, you are creating little mini-entities, and you must equip them. All this takes code, which needs test harnesses, and documentation. This takes a lot of people. So they need to communicate with each other, and coordinate, and have their work reviewed.

The seduction of layered complexity in software: an example using Ruby

For experimentation or rapid prototyping, it can be helpful to have the path smoothed by libraries, to get something up and running quickly. But the software lifecycle is long, and one must think about building out additional capabilities and/or making changes later. It is inevitable that one will have to get beneath the abstraction layers to get things to work. So the choice of application layers and libraries does not avoid the pain, merely delays it. If you understand that, you can make the right decision about how much to build your product on top of a set of libraries, vs. use the libraries to test your ideas and then figure out how much of their capability you really need/want in your application, and whether it is better to write a small, tight, module for your own application that goes directly to the fundamentals, or to integrate layers. The less you incorporate the less you have to worry about a library breaking through updates. Fewer dependencies means longer code life.

Let’s take a look at an example of the down-sides. Suppose you were wanted to parse a web-page’s HTML tags to extract some information, something quite common in the world of site scraping.
A little Googling, or ChatGPT for coding, can give you readily these a few lines of Ruby code to achieve this:

create Browser.new
page = Browser.url(x).getpage()
fields = page.parse
fields.foreach field do field==y if file.write(field) then end

Clean, simple, easy enough to get something working. But what’s needed behind these lines?

require url
require get
require parse
require browser

The next bit is parsing. The document returned is an HTML doc with a tag structure, and you’re looking for certain tags. That gets a bit fiddly, as you debate whether you need a robust tag parser or roll your own. A robust tag parser is easy enough to hook up, a few more layers and libraries that do various bits. And so it goes on.

Each package (module) brings with it abstractions, libraries of methods, and hidden complexity.

Then there’s the compatibility and upgrade considerations, the changes to the language itself. Suppose this was Ruby 1.9 code with Ruby 1.9 libraries. Three years later, Ruby 2.0 would present a choice as it incorporated changes that broke backward compatibility.
If you upgrade to the new version of the language, you will need to understand the major and minor changes, fix any breaks in your code, and test comprehensively. If you hold out, then you are running your code-base on libraries and code that that are no longer maintained, upgraded, checked. You start to accumulate conditional compilations and version detection for various libraries:

platform <= 1.9 if ... use  first version ... else ... use second version ... then

Another three years and Ruby 3.0 rolls out, breaking backwards compatibility with Ruby 2.0. Again the same decisions: upgrade or add to the compilation conditions:

platform <= 1.9 if ... use version 1 ...
   else platform < 3.0 if ... use version 3 else use version 2 then

Three years, and you decide to switch to Python. Now you have to sift through through Python's libraries and rebuild the layers with new libraries.

The above is over-simplified, but it illustrates the point.

What's the alternative? Getting under the hood, and grappling with the lower layers directly. Web-servers which export the get method, HTML streams with tags, TCP/IP, some thought about endian-ness of the streams and the OSes involved. By thinking through the design decisions intentionally, you may find you don't need or want all the libraries and layers. And the benefit to bypassing the layers is that you may end up with a much smaller software footprint. The point is not to just fall into incorporating layers and libraries into your design because it's easier.

8. Conclusion

Use libraries by all means. Use them for prototyping, use them for demonstrating concept. But then step back and build it right, and at the right level of abstraction. Make the use of software layers an intentional choice, not the result of a fad. If you want your code to stand the test of time, the ideas have to be crystal clear, and they should have some element of the implementation, so that they survive as long as the underlying implementation.

The problem of code bloat, and layers of code come with a short-term view. The total cost of software maintenance vs. feature buildout. If you use someone else's libraries, you are importing all of their problems and reliability and make these your problems and your reliability. In software, unlike in finance, paying later is NOT better than paying now. Paying later is more painful: time has passed, developers have moved on, documentation has become obsolete, knowledge is lost, context has vanished, including the understanding of why that particularly weird line of code was added.

The world of software development will not change dramatically, but with the right mindset, the courage to pick the right point in the hardware-software stack from which you will build, these will be the critical decisions. Your choice becomes: is it better to go a little deeper and craft what you need from a minimal subset of what was in the library, or to throw in the library and all it brings with it.

If you've understood value of layers in speed/exploration of capability but also the importance of simplicity and avoidance of excessive dependencies, then you're well prepared to go on and do great stuff! Conceive new ideas and applications! Seek simplicity of the overall hardware-software combination. And remember that every external piece of code you pull in to your work is code that you will need to understand, and that you will need to take along for the lifetime of your application. Be wise. Think long-term.

---

Appendix 1. Assembly Language in Embedded Systems

There are at least four examples in embedded systems development in which some knowledge of assembly language programming is useful:

1. for platforms that do not have C compilers, assembly language is usually the default programming interface. For many years, PIC microcontrollers¹⁴ under $3 used to be without a C compiler due to their extensive array of built-in peripherals and idiosyncratic architecture (without a nestable parameter) making it more difficult to write a generic compiler. Nowadays, the notorious 10-cent microcontrollers and other sub-$1 MCUs tend to have toolchains that work directly with assemblers and do not have targeting compilers.
2. when systems require hard real-time performance in the absence of a real-time operating system (RTOS), assembly language on a RISC (reduced instruction set computing) chip allows interacting directly with the silicon and being able to obtain accurate timing characteristics by counting instructions since in a RISC chip almost every instruction takes a fixed and uniform time for the CPU to execute.¹⁵
3. when writing device drivers or obtaining peripheral access in the absence of an operating system, or when writing drivers for an operating system.
4. when it is required to directly supervise the storage of values to specific registers or the use of specific elements of an underlying CPU architecture, C is often unable to help. Being able to drop into assembly, or Forth either inline or through functions, is then the most direct method of access.

---

Related Readings List

1. [Hoare, 1981] C.A.R. (Tony) Hoare, The Emperor's Old Clothes, p.9, Communications of the ACM, 1981
2. [Lee, 1996], John A.N. Lee, "Those Who Forget the Lessons of History Are Doomed To Repeat It” or, "Why I Study the History of Computing. IEEE Annals of the History of Computing, 1996, Vol.18, Issue.2, pp.54-62. (Online - membership required)
3. [Moore, 1999], 1x Forth, an Interview of Chuck Moore (Online)
4. [Spolsky, 2002], Joel Spolsky, "The Law of Leaky Abstractions" (Online)
5. [Daylight], Dijkstra as a Window into Computing History, by Edgar Graham Daylight
6. [Dijkstra], life and work, the view from his shoulders
7. [Daylight, 2010], Edgar A. Daylight, Dijkstra's Rallying Cry for Generalization (Online)
8. [Wirth, 2008], Niklaus Wirth, A Brief History of Software Engineering (Online - membership required)
9. [Controversy in Algol's History], How Recursion got into Programming through a Sneaky side door into Algol
10. Algol 68 decides to pass on the chance to be Pascal, with Dijsktra, Hoare, and 6 others dissenting in a Minority Report declaring Algol 68 a failure (note that by 1968 Dijkstra and van Wijngaarden had fallen out)
11. At least through 1975, Dijkstra was recommending his students learn Pascal.
12. [Basili, 2002], Lessons Learned from 25 years of process improvement: the rise and fall of the NASA software engineering laboratory, IEEE, (Online - membership required)
13. [Fox, 1998] Jeff Fox, Low Fat Computing (Online)
14. [Graham, 2003] Paul Graham, "A 100-year Language", (Online)
15. [Morris, 2012] Richard Morris, The Lost Art of Keeping It Simple: An Interview with Charles (Chuck) Moore, (Online)
16. [Williams, 2017] Elliot Williams, Forth: The Hacker's Language, Hackaday (Online)
17. [Haydon, 1991] Glen Haydon, Levels of Forth (Online)
18. [Gentle, 2015] Sam Gentle, Less is Moore? (Online)
19. [Moore, 1970] Chuck Moore, Programming a Problem Oriented Language (Online)
20. [Sergeant, 1991], Frank Sergeant, 1991, "A Three-Instruction Forth" (Online)
21. [Gentle, 2015], Less is Moore, by Sam Gentle, 2015

Related Articles on this site:
Ubiquitous Computing - Knowledge Engineering & Emerging Technologies: A View of what's coming in the Next Decade (2005-2015)*

The Software Stack
Software as a Force Multiplier, Sections 1-3
Seven Fields of Computer Programming: A Brief Survey
Language-Oriented Programming: Forth, Lisp, and Ruby: Languages that enable solving your problem in a language appropriate for your domain (domain-specific language)
The Forth Language for (Embedded) Systems Programming
Bare Bones Programming: The C Language for Embedded and Low-Level Systems Development
A low-level toolchain for x86: NASM (Netwide Assembler) and TCC (Tiny C Compiler)

The Embedded Stack - Microcontrollers, Sensors, and Embedded Systems: Low-Cost Experimenting with Arduino

Curated Shorts - Advances in Emerging Technology

---

Endnotes