FreeBSD installation and package tools, past, present and future

Title: FreeBSD installation and package tools, past, present and futureDate: September 17th, 2000Author: Jordan K. HubbardVersion: 1.2Abstract:This document discusses FreeBSD‘s installation, configuration andpackage management tools from the perspective of where they are andwhere I think they need to go.Contents--------1. Preface2. History and current limitations2.1 The package tools2.2 Sysinstall3. The Future3.1 FreeBSD‘s distribution format3.2 User Interface3.3 Security3.4 Configuration and version control3.5 Installation scripting4. Appendix: Current efforts4.1 libh4.2 lizard1. Preface----------There has been a lot of discussion throughout FreeBSD‘s history as tojust what purpose sysinstall and the pkg_install suite were intendedto achieve, what their shortcomings are and how we might move forwardwith a design document which breaks the various challenges into moremanageable pieces which might be implemented by a number of differentteams.It‘s long been my desire to sit down and do exactly that, a lack oftime being my only excuse for not having done so long ago. I‘m alsoof the understanding that a new "open packages" effort was recentlystarted by some of the people at Daemon News, a project with parallelsto some of the existing efforts to get all the various open sourceprojects to standardize on existing package formats like RPM, Debianpackages, etc., and a good excuse for me to finally do this.I‘m certainly all in favor of a standardization effort based aroundsome viable and practical second-generation technology and can onlyhope that producing this document will in some way aid the design of anext-generation package and installation system. Should such aneffort ultimately prove itself attractive to a large segment of theopen source community then all the better, but we have to startsomewhere and that somewhere, for me at least, is FreeBSD. Theexisting package systems (RPM, Deb, *BSD) all suffer from beingfirst-generation efforts and, while quite mature, do not address anumber of significant issues which I‘ll cover in this document. I‘llalso document some of the design decisions which went into FreeBSD‘scurrent system, hopefully explaining some of the [mis]features whichhave confused newcomers to FreeBSD or caused them to wonder just whythings were not done differently.2. History and current limitations----------------------------------2.1 The package tools---------------------The FreeBSD package tools, located in /usr/src/usr.sbin/pkg_install,were written in August of 1993 in response to several requirementsthat we had at the time. Most significantly, it was not possible toeasily track "extra software" that one might add to the system andconceivably wish to easily remove again, nor was it easy to see whichversions of software had been installed on a given system for easiertroubleshooting. Finally, any specialized installation procedures fora given piece of software essentially had to be done manually byreading the README file (when available) accompanying the binarydistribution tarball, assuming of course that anything other thansources which you needed to build yourself were available.After looking at the problem for awhile, I decided that the quickestand easiest solution would be to simply add a little extra "meta-data"to these existing binary tarballs, something which could then beexecuted and recorded for future reference by a package addingutility. Thus were born the pkg_install utilities we have today.At the time, system administrators were also very mistrustful ofpre-built binary distributions of software (not that many wouldactually read source code before building and installing binaries fromit, but that‘s another story) so that‘s why I decided to use anexisting archive format, namely gzipped tar files. This approachallowed paranoid admins to easily extract a package manually andinspect it, it also allowing me to leverage our existing toolsrelatively easily (though one feature, --fast-read, did need to beadded to tar so that individual items could be extracted morequickly).There were and are problems with this approach, however, the mostsignificant being that tar files (especially gzipped ones) are NOTvery amenable to random-access. The directory structure of a tarfileis distributed, e.g. the file data is interleaved with the directorymeta-data and, in order to get to a given item in a tarball,pkg_add(1) needs to read serially through the whole thing looking forit. This can be an especially big problem when all it has to workwith is a file handle and not an actual file, something which is thecase when a package is coming directly from an FTP server or someother data source which offers only serial access to the bits.pkg_add "solves" this problem by first finding sufficient temporaryspace on one of the available file systems and then unpacking thetarball to be extracted into a scratch directory. After the tarballis extracted, pkg_add then reads through the "packing list" (one ofthe meta-data files) and follow its instructions to move only thoseparts of the unpacked tarball into place which are needed, thusskipping the meta-data files and any others which might be optionaland not actually requested by the user. During this process, it isthen possible to run any custom installation scripts the package mighthave provided to ask the user configuration questions, do specialpermissions/conflict checks, and run through the package‘s list ofdependencies on other packages to see if they should be somehowfetched and installed as well.All in all, it‘s a very general purpose and open-ended mechanism whichmany packages have used to good effect, but the temporary directoryrequirement would also turn around to bite me firmly on the ass whenit came time to write sysinstall, which followed in April of 1995.2.2 Sysinstall--------------Sysinstall, located in /usr/src/release/sysinstall, was FreeBSD‘sfirst attempt at doing something more elegant and user-friendly than asimple shell script-based installation which merely asked questions ina fixed order and gave the user little opportunity to do differenttypes of installation and configuration. The "first draft" ofsysinstall was actually meant to be little more than a prototype ofthe installer I really wanted to write, especially from the userinterface perspective since it used something called dialog(3). Thedialog library began its life as a monolithic utility for writingsemi-graphical shell scripts and was pressed, with great reluctance,into the duty of functioning as an interface library for Cprogrammers. At the time, this seemed the easiest course of actiongiven that I wasn‘t overly keen on writing a new set of interfacecomponents in curses(3) and the dialog library provided some fairlycolorful canned dialogs which looked, at least for the time,reasonably visually impressive.In retrospect, this was also one of my biggest mistakes given thatdialog(3) is also extremely limited in the user-friendlinessdepartment and lacks features like the ability to put more than 2buttons into a dialog or a Yes/No dialog which has a selectabledefault (e.g. No). The inability to put a "Back" button into variousdialogs which could really use one or the necessity for asking only"positive" questions are outgrowths of those limitations and goodexamples of how an insufficiently powerful UI library can drive theutility-writer in undesirable but unavoidable directions.The dialog library also features checkbox/radio menus which use thespacebar and enter keys very, erm, creatively to essentially confusethe heck of out users who don‘t pay too much attention to the Usageinstructions at the beginning and simply impulsively hit Enter throughthe whole installation. Earlier versions of the library alsocompletely lacked the idea of call-backs, so any form of real"dynamism" in a menu or dialog was pretty much out of the question.The things I had to do to this library in order to provide thosefeatures were so hideous that I‘ll probably go to a specialprogrammer‘s hell when I die and be forced to do AI programming inRPG-II, or something, it also souring me on the idea of extendingdialog(3) to the point where it might have actually made sysinstallless pathological in its interface behavior.The user interface library has also turned out to be not the least ofsysinstall‘s design shortcomings. Since it was, at least in my mind,a prototype, there wasn‘t a lot of attention put into the area offlexibility. I provided for things like "Expert" and "Novice" (nowless-insultingly named "Standard") installs, but I didn‘t really domuch for people who wished to build many machines in a moreassembly-line fashion or allow the user to save their answers to itsquestions for later "replay" into another installation session.Extending sysinstall also requires a knowledge of C programming (andthe willingness to hack on a prototype) in order to customize it forother purposes, say a university environment where special course-waremight be part of the FreeBSD installation at the beginning of eachsemester. It‘s nowhere near as easy as it should be and many havebeen impaled on sysinstall in their efforts to customize FreeBSD fortheir unique needs.An even more significant issue with sysinstall and FreeBSD‘s releasemethodology in general is the distribution format of FreeBSD itselfand sysinstall‘s handling of packages, especially interactive ones.FreeBSD‘s release methodology has really not changed all that much inthe last 8 years, the basic distribution format still being largelyinfluenced by the size of a 3.5" floppy. Each chunk of a FreeBSDdistribution, e.g. the "bin" or "manpages" distributions, is nothingmore than one big gzipped tarball which has been split into 240Kchunks which can conveniently fit on floppies, 5 to a 5.25" floppy or6 to a 3.5" one. Back in 1992, when we first started doing this,there were a lot of people doing floppy installs and CDs were stilluncommon and/or expensive. Sysinstall was therefore designed to takea lot of the hair out of the process by automagically gluing these240K chunks together as they came along, from whatever distributionmedium was available, and feeding them to a background tar processwhich would simply extract them verbatim into a directory (usually,but not always, /).There are lots of problems with this, one being the fact that since a"distribution" is nothing more than a gzipped tarball split intopieces, there is none of the nifty meta-data which packages provide tosay what has been installed, what dependencies it has, or any hooksfor providing post-installation configuration opportunities. Evencomponent size information is a mystery, making sysinstall unable topredict when you‘ve chosen more distribution data than will fit on agiven filesystem, leading to occasionally unpleasant surprises duringinstallation when something fills up and simply exlodes in a messy andunhelpful fashion.A bigger problem is the fuzzy and entirely undesirable dividing linebetween packages and distributions. What should be a distribution andwhat should be a package? Where does the ``base distribution‘‘ stopand the ports/packages collection begin? How should one upgrade therespective bits? Erasing this line of demarcation has proven to beone of the more annoying challenges in FreeBSD‘s release engineeringprocess and I‘ll explain how and why later in this document.Finally, sysinstall simply represents a conglomeration of too manytasks. It partitions your disk(s), it loads software, it asks youquestions about your network interfaces, it sets up your pppconnection, etc etc. It just tries to do too much in one place andthat‘s a violation of the Unix Philosophy, where each component shoulddo one easily recognizable task and no more than that, more complextasks being achieved by putting such tools together.What we currently think of as sysinstall should essentially do nothingmore than partition your disks and get a much fancier second-stage"configurator" onto the root partition before rebooting. At thatstage, the configurator can give the user the option of adding theother disks and chosing what kinds of software to put on them. Thescope of the configurator should be such that it becomes ageneral-purpose setup tool which can be used to manage all thehardware and software in the system on an ongoing basis, not simplyrun once and forgotten.3. The Future-------------3.1 FreeBSD‘s distribution format---------------------------------As I mentioned in the history section, one of the more annoyingproblems with FreeBSD‘s current distribution format is the dividingline between distributions and packages. There should really only beone type of "distribution format" and, of course, it should be thepackage (There Can Be Only One). Achieving this means we‘re firstgoing to have to grapple with several problems, however:First, eliminating the distribution format means either teaching thepackage tools how to deal with a split archive format (they currentlydo not) or divorcing ourselves forever from floppies as a distributionmedium. This is an issue which would seem an easy one to decide butinvariably becomes Highly Religious(tm) every time it‘s brought up.In some dark corner of the world, there always seems to be somebodystill installing FreeBSD via floppies and even some of the fortune 500folks can cite FreeBSD success stories where they resurrected some old386 box (with only a floppy drive and no networking/CD/...) and turnedit into the star of the office/saved the company/etc etc. That‘s notto say we can‘t still bite that particular bullet, just that it‘s nota decision which will go down easily with everyone and should be wellthought-out.Second, there‘s the issue of packages currently requiring temporaryspace as part of their extraction method. If we‘re going to havethings like "bin" be a package, even if we split it up intosubcomponents and make "bin" simply a package which contains a list ofdependencies and nothing more (which is desirable), there are stillgoing to be pieces which are non-extractable under the current schemebecause the available disk space is too small to contain both thetemporary copy and the final installed copy, which may not be on thesame file system and cannot be simply moved into place. Since we‘dalso like to retain the ability to extract a package directly over anetwork connection and never have the temporary bits "hit the disk",this means that we‘re almost certainly going to have to go to adifferent archival format. Fortunately, there are some existingformats to choose from which have a lot of the required features so wewon‘t have to reinvent the wheel and come up with our own (yuck). Mycurrent favorite is the Zip archive format.Zip is a popular archival format which gives us a wide variety ofexisting tools for creating, fixing and inspecting zip files. Thedirectory is also at the very beginning so we can quickly read it inand figure out where in the data stream/file we need to go to get aspecific item. Since the "configurator" stage of the installationwill also be running after we‘ve acquired a root partition and someswap space, it‘s also not inconceivable that we could buffer bits readover a network connection in memory so that even "seeking" out to theend of an archive file read from an FTP server socket would stillallow us to move backwards in the archive for other contents. The zipfile format also allows for per-archive and per-file "comment" fieldswhich can be used to store things like MD5 checksums, pgp signaturesand all sorts of other potentially useful types of meta-data. I‘m notwedded to the zip file format, I simply find its combination of goodcompression and random-access (without having to decompress the entirearchive) to be especially attractive for what we need to do.Finally, there‘s the issue of user interaction. The bulk ofsysinstall‘s hard-coded features do things like make user querieswhich could just as easily be part of a package‘s install-timeconfiguration script. Sysinstall, for example, allows you to specifywhich daemons will run at startup time even though this is onlypertinent to the "bin" package which actually contains those daemons.Similarly, there have been security-related questions pertaining tothe cryptography distributions which, even though the US crypto exportand RSA issues have now been largely dealt with, may still beapplicable in other countries. Clearly, such interaction should bepart of the package installation procedure itself and sysinstallshould be little more than a friendly wrapper for selecting whichpackages to install and running their installation procedures, andthat brings us to the question of User Interface.3.2 User Interface------------------As noted in the History section, one of the biggest problems withsysinstall is its user interface which could only be charitablydescribed as Evil Incarnate. The dialog(3) interface library, as I‘vealready described, is insufficiently powerful to give the user aflexible and intuitive installation experience nor it does not takeany real advantage of environments like the X Window System, shouldthe user be running a configurator under such an environment.The package system also suffers significantly in the UI area since thepkg_add(1) utility has no idea as to whether it‘s running at the endof a pipe, as it currently does under sysinstall, or if it‘s got areal live user at the other end who‘s invoked it interactively from ashell. This leads to real problems when a package suddenly decides itwants to talk to the user but is being run via a front-end which willreact adversely (or not at all) to the sudden appearance of thepackage‘s own interaction dialogs. This is not just a hypotheticalsituation but one which can, and currently does, happen wheneversysinstall‘s packages menu invokes a package which is interactive. Theuser dialogs all go to the 2nd VTY and leave the actual user somewhatmystified as to why the package installation has mysteriously "hung"on them as it waits for user input which never arrives.To effectively solve this problem, what is needed is a flexible (e.g.containing more basic "widgets" than canned dialogs) and generic UIlibrary which provides front-end utilities like sysinstall and pkg_addwith the ability to play traffic cop and direct all user interactionthrough a common interface. That might be something CUI based, likeTurboVision (my current CUI favorite) or GUI based, like Qt/gtk ifrunning under X. It might even be something which talks to aJava-enabled web browser at some point in the future - we really can‘tpredict all the conceivable UI scenarios. The package system wouldcall into this library whenever it wanted to talk to the user, thussharing the screen/display non-competitively with whatever utilityinvoked it. It would be up to the outermost "caller" (be it pkg_addor sysinstall) to decide at initialization time just what kind ofback-end UI to instantiate for the generic UI.Such an approach would allow us to write all of our configurationutilities and scripts in a UI-neutral fashion which allows us to takeadvantage of new UI technologies as they come along without having togo back and re-write all of those painstakingly crafted user dialogs.That‘s basically where 99% of all the work of crafting such userinterfaces goes, and we certainly don‘t want to have to write twodifferent interface definitions for CUI (serial console / remoteinstaller) and GUI (X Desktop) based users. There are some operatingsystems (that I won‘t mention) which sort of get away with this today,but FreeBSD has always been a strongly server-centric operating systemand that means we really can‘t have a highly desktop-centricinstaller, we have to support the idea of installation on machineswithout graphics cards at all or even in situations where the user isvisually handicapped and wishes to have a customized installer who‘s"interface" is a voice synthesizer. All of this is possible when theUI library you write directly to makes no assumptions at all aboutwhat the ultimate rendering model is going to be, it simply thinks interms of objects like "buttons" and "choice lists", leaving it up tothe back-end layer to ultimately render the appropriate UI objectssomehow.3.3 Security------------A major failing of most package systems, ours included, is that apackage‘s installation and configuration scripts can essentially beany type of executable at all. While this does allow the packagewriter a great deal of flexibility in providing for a package‘s needs,and there are packages which do have highly specialized requirements,it also has a huge potential effect on security.Most packages are installed as root for a variety of reasons, somelegitimate and some not, and the overall effect is that security isessentially an "opt-in" process for whomever creates or installs apackage. A package which is installed as root is a package which canbe either intentionally or unintentionally lethal to a user‘s system,even a pgp-signed and triple-authenticated package being capable ofcompletely destroying a user‘s system and it‘s not hard to see how.Consider what might happen if an otherwise perfectly respectablepackage author, overly caffeinated and partially delirious at 4am,were to write: ``rm -rf /${MYTMPDIR}‘‘ into a package‘s installationscript as part of its clean-up procedure. Let‘s also say that thisremoval operation was inside a failure-case check in the installationscript and the author didn‘t hit that case during their testing sincethey happened to drive the installation successfully each time. Let‘sfinally say that the actual name of the variable in question was"MYTEMPDIR" and the author, in a state of 4am dyslexia, did not spotthis mistake. You get the idea.Even if the package is pgp signed and the package author is yourpersonal, trusted friend, you‘re still going to be wondering at allthe sudden extra disk activity right after bombing out of hispackage‘s installation script and none of the conventional securitypractices will have saved you from his mistake. The author is mostembarrassed, your system is most toast, and you can both chalk it upto another annoying conjunction of human and infra-structuralstupidity. Clearly, it would be desirable for a package whichgenuinely and truly needs to be root to do so in a manner which is inany way safer than it is now.One method I‘m in favor of is to change a package‘s customizationscript(s) from being any arbitrary executable to being a very specificexecutable, namely a set of instructions in some tightly constrainedscripting language. My personal favorite is Secure TCL, a usefuloutgrowth of the enhancements done to TCL when it got stuffed into aweb browser and suddenly needed to worry a lot more about securityand trust issues. Secure TCL allows us to create highly restricted TCLenvironments which can be selectively "tightened" according to anadministrator‘s own level of paranoia, allowing them to have a highlycustomizable and final say over what level of capability will be givento any package they install.Thus it would be possible, just to give an example, to restrict the``file-access‘‘ primitive to only returning a positive"It‘s OK to access this" indication for file names who‘s paths match"/etc/.*", "/usr/local/.*" or "/usr/X11R6/.*". The ``file-create‘‘,``file-write‘‘ and ``file-remove‘‘ primitives could, in turn, alwaysvalidate their arguments against ``file-access‘‘ before proceeding.With a properly designed set of primitives, it would be thus possibleto evolve mechanisms for "practical security", where potentiallyfoot-shooting primitives can either be disallowed entirely, allowed toproceed only upon user confirmation or go completely unhindered, allaccording to the administrator‘s wishes. With a little time, suchpackage security tweaks would also begin to float around and comeinto the reach of less skilled administrators, just as standardizedcisco access-lists for fire-walling are passed around today.It need not be TCL that is chosen for this purpose, naturally, it‘ssimply my personal preference since I happen to already know and haveworking experience with TCL. A language like Python or Ruby is alsoprobably capable of doing the job just as well, it only beingnecessary for the interpreted language of choice to have some sort ofreasonable security model and a comparatively small footprint. Istipulate that the footprint needs to be small because any futuresystem configurator and package infrastructure will need to be wrappedtogether to some extent, the resulting product being something we maywish to bootstrap off of comparatively small media. A properlywritten package management system will be an indispensable piece ofthe installation process given that the pieces of the operating systemwill, of course, be packages.3.4 Configuration and version control-------------------------------------Ultimately, installing the "OS networking package" or the "ApacheServer" package should be part of a seamless, "one piece" installationexperience with a common and consistent UI. The ability to leave"configurators" for each subsystem or tool behind should also be anintegral part of the process, these later being runnable from asingle front-end tool (let‘s call it ``setup‘‘) which offers aproperly organized menu/folder hierarchy for all the available toolconfigurators to drop themselves into. None of this is rocket scienceand folks like Microsoft and Apple have been doing it for ages withtheir operating systems. It‘s a workable model and, perhaps moreimportantly, it‘s now the most familiar model.Another nice thing about having a package install itself through acarefully controlled scripting language is that each mutagenicoperation (say, a file overlay) can store "undo" information foritself if given enough available disk space. Imagine that all ofthe undo information for a given package, throughout its lineage, goesonto an "undo stack" for that package. If necessary, the package canthus be "popped" back through its previous versions to test and seewhere and if a given problem (which may be noticed only months afterthe last upgrade) first appeared. Since the changes would be storedas deltas, files which do not change would also appear only once andno space wasted in representing multiple redundant copies of thosepieces of a package which don‘t change from version to version (likethe docs :-).Making such a mechanism part of the basic infrastructure may strikesome as an over-kill proposal, but I would also submit that theproblem of upgrading packages and of having multiple active versionsof a single package (like gtk or TCL) are significant issues whichhave received rather ad-hoc attention to date. With the creative andautomated use of symlinks, some filename hashing, and a file versionstore, I think we could come up with a mechanism which does for packageversion control what CVS does for software version control (thoughhopefully even less painfully :).Since the operating system itself would also be a set of packages,the whole idea of OS updates (especially the kind one can easilyback-out in case of emergency) become especially easy to deal withand generally more powerful than what most systems offer as well.Remember, disks are only getting bigger and cheaper, and for somepeople the extra security of a tall "undo stack" may be well-worthsacrificing a gigabyte or two to the cause. For those on tighterstorage budgets, the size of the "undo area" can be configured muchsmaller or eliminated entirely.A genuine database of some sort containing package version meta-datais also a requirement since, on a fully tricked-out system, manyhundreds (if not thousands) of files might eventually be involved andkeeping track of various their inter-relationships is not somethingyou‘d generally want to do with simplistic file structures (like/var/db/pkg) which require a lot of time to search and index.3.5 Installation scripting--------------------------Another subject I touched on earlier was the need for automated and/orhighly customized installations since the needs of everyone installingFreeBSD aren‘t exactly identical. Given access to a nice generic UIlibrary, as described in section 3.2, and a powerful scriptinglanguage, as described in section 3.3, we could make what peoplecurrently regard as sysinstall a purely script-driven affair. Thiswill obviously make customization a lot easier since all anyone needsis a text-editor and a document of available primitives (which manywould probably choose to learn simply by looking at the exampleinstallation anyway) in order to create a customized install and/oradd their own questions to an existing package configurator. I alsodoubt that most people would need to be able to do this, but for thosevery few that do, such flexibility can and will make the differencebetween getting FreeBSD into some highly customized environments orsimply not making the grade.4. Appendix: Current efforts----------------------------4.1. libh---------The libh project is something I started over a year ago, with inputfrom Mike Smith and the paid services of a Russian contract programmernamed Eugene, to fulfill many of the goals expressed in this document.Unfortunately, managing a project of this complexity with a contractormany thousands of miles away and a personal schedule which allowed forvery little interaction with him didn‘t prove to be a workablescenario and work was stopped while partially in progress. Since thattime, work on it has been taken over by Alexander Langer and a smallgroup of volunteers. A mailing list, freebsd-libh, can also besubscribed to via majordomo@freebsd.org, and the sources checked outvia ``:pserver:anonymous@usw4.freebsd.org:/home/libh/cvs‘‘ usinganoncvs.The name ``libh‘‘ is also something of a mystery to everyone but itnonetheless stuck as a working title. It probably needs to be renamedto something sexier before this project can really succeed. :-)Roughly speaking, libh currently contains:A first cut at the generic UI library, as described in section 3.2,with back-end renderers for TurboVision and Qt currently beingprovided. The generic UI API it provides is available for C, C++and TCL.A complete zip file-access library written for C, C++ and TCL asdescribed in section 3.1.Much of the security infrastructure described in section 3.2 isalso implemented, with enough currently done to make possible aprototype package creation/extraction system with some testpackages available (and used as part of the regression-test suite).The package information database is also written, with APIs for C,C++ and TCL. It provides for package conflict, upgrade and outdatechecking.While libh does contain a lot of the code we might ultimately use, itshould nonetheless be considered only one possible starting point forimplementing what I‘ve described in this document. I certainly wouldbe happy to see the time and investment in libh ultimately go to gooduse, of course, but I also wouldn‘t want it to stand in the way of anylarger and more successful effort which chose a different scriptinglanguage or UI design, for example.4.2 lizard----------Lizard is the installer currently bundled, albeit in highly modifiedform, with Caldera‘s OpenLinux distribution and made freely availablein some of its earlier incarnations from ftp.caldera.com. It has beensuggested that a "Desktop version" of FreeBSD could be created usingthis technology as a stop-gap measure until libh or some similarproject succeeded in solving the more complex set of issues I‘veoutlined, that perhaps buying us a bit more time to "do things right"(in my highly prejudicial opinion :). As far as I‘m aware from mylimited reading of the code, lizard is only applicable to graphicalinstallations and does not make allowances for people installing via aserial console, hence its applicability to just a desktop-orientedproduct. Still, it might be worth looking at by people who‘sinterests lie solely in that direction. Customization from the highlylinux-centric environment lizard currently assumes is, of course,something else which would need to be grappled with as part of such aneffort.