On the side, I’ve been wrapping up some improvements to the classic Unix
stdio libraries in illumos. stdio contains the
classic functions like fopen(),
printf(), and the security
nightmare gets().
While working on support for
fmemopen()
and friends I got to reacquaint myself with some of the joys of the
stdio ABI and its history from
7th Edition Unix. With
that in mind, let’s dive into this, history, and some mistakes not to
repeat. While this is written from the perspective of the C programming
language, aspects of it apply to many other languages.
APIs and ABIs
Before we dive into a discussion of the stdio ABI, let’s first talk about APIs and ABIs. In programming, many people talk about application programming interfaces (APIs). APIs define how a program can call into another piece of functionality. In C, APIs are often contained in header files and these are documented in manual pages. For example, here are some API declarations in C:
extern int fstat(int struct stat *);
extern void *(size_t);
extern int fputc(int, FILE *);The API names a function and describes the types of the parameters and the return value of the function. These declarations (and dependent headers) are all that one needs to write a C program. While the API is enough to write a program, when the compiler and link-editor get to work and you actually run your program, you need to rely on something else entirely: the ABI.
The
application
binary interface (ABI) describes a lot of aspects of the program that
are required to have it run. For example, when you call into libc, where
are the arguments found? Are they found on the stack? Are they found in
registers? The ABI also describes certain things like how many bytes
comprise an int and how should one lay out a structure.
Now, a large amount of the ABI is standardized in different documents for a given platform. For example, many Unix-based systems follow the System V ABI. This ABI defined many aspects of systems such as ELF (executable and linkable format) and dynamic linking. Portions of the ABI were relegated to processor-specific parts. These describe the use of registers, the alignment of types, and more. Some of the more interesting documents include:
These documents don’t describe everything that one needs. For example,
while Linux, illumos, and the various BSDs all use the same amd64
calling conventions and ELF-based binaries and libraries, there are many
things that each system uniquely defines. For example, let’s compare the
struct stat used in illumos and OpenBSD. I’ve placed the two side by
side, though I’ve trimmed out a bunch of pre-processor macros:
illumos | OpenBSD
|
struct stat { | struct stat {
dev_t st_dev; | mode_t st_mode;
ino_t st_ino; | dev_t st_dev;
mode_t st_mode; | ino_t st_ino;
nlink_t st_nlink; | nlink_t st_nlink;
uid_t st_uid; | uid_t st_uid;
gid_t st_gid; | gid_t st_gid;
dev_t st_rdev; | dev_t st_rdev;
off_t st_size; | struct timespec st_atim;
timestruc_t st_atim; | struct timespec st_mtim;
timestruc_t st_mtim; | struct timespec st_ctim;
timestruc_t st_ctim; | off_t st_size;
blksize_t st_blksize; | blkcnt_t st_blocks;
blkcnt_t st_blocks; | blksize_t st_blksize;
char st_fstype[_ST_FSTYPSZ]; | u_int32_t st_flags;
}; | u_int32_t st_gen;
| struct timespec __st_birthtim;
| };
Here, while there are a number of members which have the same types, the
layout of these such as the order of the members is rather different.
There’s nothing wrong with that; however, it’s pieces like this that are
part of each system’s ABI. While these differences may seem a bit
mundane at first, when you start to go further afield from a Unix-family
system and say contrast with Windows, the differences are much larger.
For example, the long data-type in 64-bit Windows is a 32-bit value,
whereas it’s a 64-bit value in Unix-family systems.
Backwards Compatibility
You might reasonably ask, why do we care about these ABIs? Well, one of the areas where ABIs are quite important is in backwards compatibility. Backwards compatibility refers to the ability for today’s systems to run software that was built in the past. If that future system can run the older program, it is considered backwards compatibility. A common example is how you could play Gamecube games on the Nintendo Wii.
Different systems value backwards compatibility in different ways. Windows is famed (and cursed) for its backwards compatibility. One can run old software that targeted Windows 95 on Windows 10. Apple takes a different view entirely, and they generally drop compatibility after two or three major releases. Linus Torvalds long instilled the rule of "not breaking user space" in the Linux kernel, which is a similar edict about backwards compatibility for user applications, though the Linux kernel doesn’t have a stable API or ABI for drivers.
In illumos, we generally value maintaining a stable ABI over time so that way older software will continue to work. Maintaining backwards compatibility does come with some cost and sometimes requires that you put forth more effort to deal with it. However, if every major update meant you had to rebuild all of the software for your operating system, that can also be frustrating. illumos isn’t the only system with this property. A lot of other software is also known for valuing backwards compatibility. A notable example is glibc.
You might ask what does all of this talk of backwards compatibility
have to do with the ABI. The two are pretty closely related. Take the
struct stat example above. When a C program is compiled, the offsets
of the members of a structure become party of the generated binary. If a
new member were inserted into the middle of the structure that would
change the offsets of all subsequent members and would mean that older
programs would access the wrong thing.
The struct stat also shows another challenge with the ABI. If you look
at the way that a program often uses it, it looks something like:
void
foo(const char *path)
{
struct stat st;
if (stat(path, &st) == 0)
printf("%x\n", st.st_mode);
}Here, the application has declared the stat structure on the stack.
This means that the size of it is known at compile time and used. If we
increased the size of the structure, but ran a program compiled using
the older, smaller size of the structure, the stat() call would write
beyond the space allocated to the structure on the stack and wreak
havoc.
With all this in mind, there’s a lot we can learn from stdio and in particular what not to do.
The History of stdio
Standard I/O (stdio) was introduced in 7th Edition Unix. Many aspects of
this
original
version of stdio are the same today. The FILE *, fopen(), getc(),
putc(), and even stdin, stdout, and stderr were all there. The
now-common header file /usr/include/stdio.h was
introduced
as well.
From there, it soon entered into the land of BSD. The second BSD release in 1979 actually contains its own implementation of a small part of stdio. However, by the time of BSD 2.9, if not earlier, it ended up with a version of stdio that looked pretty similar to the one in 7th edition. Eventually though, the folks at Berkeley wrote their own version of it. You can see that most clearly in the 4.4BSD version.
illumos, due to its heritage from Solaris, was a combination of BSD and System V Release 4 (SVR4). An interesting side effect of this is that we can track most of the implementation back to the release of SVR4. In fact, you can even see all the original AT&T copyrights in the files. Many parts of the code look similar to the corresponding version in 7th Edition. Many of the file names are even the same!
As a result of this particular bit of history, a lot of the ABI decisions that had been made in 7th edition were carried into SVR4 and made their way into Solaris. If we look at some of the decisions with a modern eye, they’re all things that make it much harder to extend standard I/O without breaking the ABI.
The FILE Structure
At the heart of stdio is the FILE structure. We’ll look at a portion of
stdio.h from 7th Edition. This comes from
The Unix Tree:
#define BUFSIZ 512
#define _NFILE 20
# ifndef FILE
extern struct _iobuf {
char *_ptr;
int _cnt;
char *_base;
char _flag;
char _file;
} _iob[_NFILE];
# endif
...
#define NULL 0
#define FILE struct _iobuf
#define EOF (-1)
...Well, here we go. This struct _iobuf is what everyone knows and loves
as the FILE structure. A lot of this structure is still with us today. If you
look at different versions of the structure, you’ll see the same
members. The _ptr, _cnt, and _base members are used to help
implement the buffering policies. The _flag member is how the
structure kept track of knowing if it was open for read, write, or had
hit EOF or an error.
The _file member is particularly interesting. This represented the
file descriptor that was backed by the FILE structure. If you consider
a char was either a signed or unsigned byte value, this limited you from
file descriptors 0 to 127 or 255. Most folks could quite reasonably have
more than 256 file descriptors open. If you’re writing a tool like grep,
this might not be true, but if you’re writing a network application and
there’s a file descriptor per connection, having more than 256
connections is quite common.
Back in 7th Edition, the FILE structures were statically allocated.
There was no dynamic allocation of them and if we look at the bit above
there, we’ll see that all you could have were 20 different FILE
structures. In that world, having a file descriptor limit probably made
a lot more sense and when this was first written it was before there was
much networking present.
Unfortunately, the fact that these structures and the array were declared this way leaked into many different applications. Let’s take a look at the 4.4BSD version of the structure:
typedef struct __sFILE {
unsigned char *_p; /* current position in (some) buffer */
int _r; /* read space left for getc() */
int _w; /* write space left for putc() */
short _flags; /* flags, below; this FILE is free if 0 */
short _file; /* fileno, if Unix descriptor, else -1 */
struct __sbuf _bf; /* the buffer (at least 1 byte, if !NULL) */
int _lbfsize; /* 0 or -_bf._size, for inline putc */
/* operations */
void *_cookie; /* cookie passed to io functions */
int (*_close) __P((void *));
int (*_read) __P((void *, char *, int));
fpos_t (*_seek) __P((void *, fpos_t, int));
int (*_write) __P((void *, const char *, int));
/* separate buffer for long sequences of ungetc() */
struct __sbuf _ub; /* ungetc buffer */
unsigned char *_up; /* saved _p when _p is doing ungetc data */
int _ur; /* saved _r when _r is counting ungetc data */
/* tricks to meet minimum requirements even when malloc() fails */
unsigned char _ubuf[3]; /* guarantee an ungetc() buffer */
unsigned char _nbuf[1]; /* guarantee a getc() buffer */
/* separate buffer for fgetline() when line crosses buffer boundary */
struct __sbuf _lb; /* buffer for fgetline() */
/* Unix stdio files get aligned to block boundaries on fseek() */
int _blksize; /* stat.st_blksize (may be != _bf._size) */
fpos_t _offset; /* current lseek offset (see WARNING) */
} FILE;While it’s different, many things are still the same. The flags and file
descriptor grew to a short, but everything is still public, which means
the layout of the structure is in the ABI. This all leads to a few
critical issues that are all intertwined.
Lack of Opacity
If we look at the APIs that exist around stdio, all of them take a FILE
*. This is a pointer to the FILE structure. This means that the use
of the APIs doesn’t actually require someone to know the size. Even from
the beginning, the APIs that gave you access to stdio, fopen(), and
fdopen() returned a FILE * and stdin, stdout, and stderr all
referred to a FILE * pointer.
All of this tells us that a consumer didn’t actually need to know the layout of the structure. If you have a consumer where you don’t actually know the layout of a structure, then we call that an opaque structure. There are a couple implications of this. Most notably it means that an application cannot allocate the structure itself, it must ask a library to, and that you need to use functions to access the members of the structure.
While in the early days of Unix, things like binary compatibility weren’t top of mind, by the 90s, some systems started caring about backwards compatibility and it was a bit too late. An existing body of software was using those fields directly.
Many of the things that we know of as functions that were standardized
in C89 such as getc() or fileno(), actually were macros and just
dereferenced the structure members. Here’s an example of another part of
V7’s stdio.h:
#define getc(p) (--(p)->_cnt>=0? *(p)->_ptr++&0377:_filbuf(p)) #define getchar() getc(stdin) #define putc(x,p) (--(p)->_cnt>=0? ((int)(*(p)->_ptr++=(unsigned)(x))):_flsbuf((unsigned)(x),p)) #define putchar(x) putc(x,stdout) #define feof(p) (((p)->_flag&_IOEOF)!=0) #define ferror(p) (((p)->_flag&_IOERR)!=0) #define fileno(p) p->_file
What we think of today as functions were actually all macros
dereferenced the FILE structure directly. Every consumer had to know the
implementation. Take fileno() for example. It returns
the corresponding file descriptor for a FILE *. Here, it just
dereferenced the field, which means that programs that called fileno()
encoded the actual size and offset of the _file member in their
programs. The same is true of all the other members referenced
If we turn to modern implementations, the stdio functions aren’t
actually implemented as macros for the most part. Folks will just pay
the cost of a function for that opacity. However, that doesn’t actually
mean it’s safe to modify and change the FILE structure around. The
problem isn’t actually just software that people compile on their own
that could have encoded it in the past. No, some modern software refuses
to let go of the encoding and accept that these structures may now be
opaque.
A great example of this is Gnulib which is a portability library. Gnulib has actually gone back and encoded the now-opaque structures that exist into itself! It will happily reach into the structures and modify them. Here’s a header file that encodes all of the structures for a number of different operating systems from Windows, to Android, to OS X, and even Minix. At this point, an operating system maintainer can’t do too much, without risking arbitrary corruption in user programs. While it’s tempting to say who cares about someone that encodes the private interface of a once-public structure, when software breaks we all lose. Users generally don’t care about who was responsible for breakage, just that it broke and that’s not necessarily a bad thing.
For better or for worse, the stdio ship has sailed here. It doesn’t matter if Android made their version private or if the Solaris 64-bit structure was never even in a public header, because stdio once visible, some software will still use it and encode the private implementation. Even when functions were added to get and set these private members.
An important take away here is that if you are concerned about backwards compatibility for a long time, a structure that users allocate themselves is probably not the right answer. Opaque structures do give library authors the ability to really change this over time. While not all software has the same constraints, when building a new interface, think twice about whether or not the structure should actually be exposed.
Member Sizing and Padding
A related challenge with the FILE structure is the size of its
original members. Several implementations stuck with using a char for
both the flags and the file descriptor (or in some cases a short). In
the context of 7th Edition, this is a reasonable choice. However, as we
consider what the impacts of sizing here are on the ABI, it makes things
more difficult.
Because of these choices, there was a fundamental limit on the file
descriptors that can be used. While 32-bit Solaris and illumos inherited
a limit of 255 (!),
FreeBSD,
NetBSD,
and
OpenBSD
today are all still limited to a short, which on current mainstay
processors is a signed 16-bit quantity or 32767. Other systems such as
musl, DragonFlyBSD, and 64-bit illumos don’t currently have the same
constraints. In some of these cases that’s due to a useful application
of hindsight or a lot of careful
engineering.
There is always a trade-off and tension between how large to size something and the corresponding memory impact. If you’re going to implement a structure whose implementation you need to maintain in a backwards compatible fashion for a while, you’ll want to think through what those sizes should be. The size that was right in the 1980s may not be right in the 2020s and the size that’s right today maybe seen as small in the future as well.
A related thing to think is structure padding. Padding is a way to add extra members that aren’t used today but may be in the future. API designers often add a bunch of arbitrary bytes to the end of a structure so they can change it in the future without having to break ABIs. This is a technique that’s used beyond operating systems. Most structures in the NVMe specification use this technique as well. Padding is also used to make sure that members in a structure are aligned at a certain granularity such as a cache line. This is done to avoid false sharing.
The nice thing about structure padding is that software will always allocate a structure of the right size, but will ignore the unused fields. This lets older software still work with newer versions of the structure, even if they don’t understand what all the fields mean.
Arrays and ABIs
A related challenge to structure opacity is the use of arrays in ABIs.
Wait, where have we really used ABIs? While systems have managed to get
around the _iob[_NFILE] declaration with fopen(), there’s actually a
more interesting case: the definitions of stdin, stdout, and
stderr. Let’s look at a couple examples side by side including 7th
Edition and 4.4BSD (which can still be found in some of the BSDs today):
7th Edition | 4.4BSD
|
#define stdin (&_iob[0]) | #define stdin (&__sF[0])
#define stdout (&_iob[1]) | #define stdout (&__sF[1])
#define stderr (&_iob[2]) | #define stderr (&__sF[2])
Here, we can see that we’ve made an array an explicit part of the ABI. This is another way that enforces the size of the existing structures. Using this technique forces the structures actually to have a known size, even if the actual member layout is private. This can be seen on various systems, such as 64-bit illumos.
If one were to do this today, they’d instead define these macros as
pointers to a symbol that existed in libc. Instead the program has a
base reference to the _iob or __sF symbol and then knows the memory
offset to find the next values at. If we were to increase the structure
size, suddenly a program that used to refer to stdout would actually
be finding a bit of the memory at the end of stdin! Trust me, having
made the mistake in development, that this doesn’t end well at all.
If you’re designing something new, think twice before an array becomes part of an ABI. And if not for this, then because of copy relocations.
ABI Lessons
To summarize, if you’re working on something where backwards compatibility is important, here are some lessons we can take from stdio:
-
Keep your structures opaque if you can! Folks can and will encode public things even if you later make them private.
-
If your structure will be public, think carefully about the sizing of members.
-
Make sure to add room for expansion in public structures.
-
Think twice before exposing an array!
Further Reading
If you want to learn more about the implementation of stdio in illumos, as part of my recent work I wrote up a design document. Parts of this will generalize to other systems.
Here are historical versions of different implementations of stdio:
Release |
stdio.h |
Implementation |
7th Edition |
||
2BSD |
||
2.9BSD |
||
4.4BSD |
||
System III |
Final Thoughts
Reading this may lead to some despair: 'Ugh, parts of this ABI are terrible.', 'Really, some things are limited to 256 file structures?', 'Growing the structure requires jumping through hoops?', 'Why do you still live with it?'.
Trying to answer the question of when you should break an API or an ABI is a tough one. The answers will vary depending on the project and what its values are. In some areas of software, the API and ABI are broken regularly. When colleagues of mine have experienced that in different frameworks, it was often quite frustrating and demotivating. Not everything has the same trade offs as an operating system and there are times where breaking the API and ABI are important and necessary.
While the constraints of working within an existing API or ABI can occasionally be infuriating, figuring out how to enable new functionality without breaking existing functionality is an important part of engineering. Studying and dealing with episodes like the legacy of stdio helps teach us what not to do. Hopefully that means that when we look ten or twenty years in the future, some of the things that we’re putting together today will have stood the test of time.
I’m sure that this won’t be the last time that I’m dealing with the design decisions whose history stretches back to 7th Edition Unix. While older code bases with 40+ years of history have a lot of rocks you may not want to turn over, there’s a lot that you can learn from them and a large number of interesting problems as well. And, if we don’t learn from them, people in 4o years will view us the same way.
Still, I find it helpful to remember what Roger Faulkner, a man who had to suffer with more of this than anyone else, was fond of saying: 'That code came from New Jersey'.
Previous Entry: Joining Oxide| All Entries | Next Entry: None