A Build System for Complex Projects: Part 3

Generating a full-fledged NetBeans build system for a non-trivial system involves multiple projects -- and more

ByGigi Sayfan
September 21, 2009
URL:http://www.ddj.com/architecture-and-design/220100417

Gigi Sayfan specializes in cross-platform object-oriented programmingin C/C++/ C#/Python/Java with emphasis on large-scale distributedsystems. He is currently trying to build intelligent machines inspiredby the brain at Numenta (www.numenta.com).

A Build System for Complex Projects: Part 1
A Build System for Complex Projects: Part 2
A Build System for Complex Projects: Part 3
A Build System for Complex Projects: Part 4
A Build System for Complex Projects: Part 5

This is the third article in a series of articles that explore an innovative build system for complicated projects. Part 1 and Part 2discussed build systems in general and the internals of the ideal buildsystem that can integrate with existing build systems. This articlewill explain in detail how the ibs can generate build files for NetBeans6.x, which is a fantastic free cross-platform IDE that supportsmultiple programming languages.

To recap: ibs is an invisible build system that doesn't require anybuild files. It relies on a regular directory and conventions to inferbuild rules and it detects dependencies automatically. It generatesbuild files for other IDEs or build systems like Makefiles, VisualStudio solutions, or NetBeans projects. It is focused on C/C++ projects,but can be extended to additional languages and other projects types.

Generating a NetBeans Build System

Back in the Hello world project (Enterprise Platinum Edition!!!) trenches Bob and the other developers picked NetBeansas their primary Mac and Linux IDE. NetBeans started as a Java IDE, butgrew in leaps and bounds to become a component-based platform forapplication development in addition to an IDE that supports all thecommon programming languages in wide use today. The latest version isNetBeans 6.7.1 and it supports C/C++ development very well and evenPython via a special early access.

The task facing Bob is to figure out the structure of the NetBeans buildfiles for C/C++ and implement the build system specific parts (thehelper and the project templates) to embrace NetBeans into the ibs.

Bob did some reading and poking around and discovered that NetBeansitself is a Makefile-based (build system generator. The way it works isthat each project's directory has a common Makefile that includes abunch of auto-generated sub-makefiles. The NetBeans user interfaceallows you to add files to every project and set dependencies betweenprojects. All this information is stored in several files that NetBeansuses to generate the proper sub-makefiles. Figure 1 shows the NetBeansIDE with the various Hello World projects.

Figure 1

The NetBeans Build System Anatomy

Let's take a look and see what all these files are about. As a runningexample Bob suggests the the 'hello' project. This is a static librarythat contains two files: hello.cpp and hello.hpp. If you are unfamiliarwith make-based build systems, you may want to take a small detour andread about it here.

The project Makefile

In the 'hello' directory there is the common Makefile. This file is thesame for every project (on a given platform). It sets some environmentvariables and contains a bunch of make targets and most importantlyincludes the project implementation makefile (generated based on theactual content of the project). Every target (a step in the buildprocess) has an empty pre and post targets that allow you to customizethe build process by executing actions before and/or after each target.The original file has elaborate comments that explain exactly whattargets are available and how you can override them. Here is an editedversion without the comments and some of the targets of the Mac OS XMakefile:

# EnvironmentMKDIR=mkdirCP=cpCCADMIN=CCadminRANLIB=ranlib# cleanclean: .clean-pre .clean-impl .clean-post.clean-pre:# Add your pre 'clean' code here....clean-post:# Add your post 'clean' code here...# allall: .all-pre .all-impl .all-post.all-pre:# Add your pre 'all' code here....all-post:# Add your post 'all' code here...# include project implementation makefileinclude nbproject/Makefile-impl.mk

nbproject

The Makefile resides in the project's directory. All the other buildfiles reside in a sub directory called nbproject. There is nothingspecial about it. It is just convenient to have all the build files intheir own directory and not cluttering the project directory. TheMakefile is the exception due to limitations of the make program.

Generated make files

NetBeans generates three "sub" make files that are included by the mainproject Makefile: Makefile-impl.mk, Makefile-Debug.mk andMakefile-Release.mk. Makefile-impl.mk is included directly by the mainMakefile and it invokes either Makefile-Debug.mk or Makefile-Release.mkdepending on the current active configuration, which is controlled bythe $CONF environment variable. In the NetBeans IDE you mayselect what configuration is active.You may also create your ownconfigurations and they will be available for activation just like thebuilt-in Debug and Release configurations with their own Makefile-your configuration.mk file. Here is the Makefile-impl.mk file of the testPunctuator project:

## Generated Makefile - do not edit!## Edit the Makefile in the project folder instead (../Makefile). Each target# has a pre- and a post- target defined where you can add customization code.## This makefile implements macros and targets common to all configurations.## NOCDDL# Building and Cleaning subprojects are done by default, but can be controlled with the SUB# macro. If SUB=no, subprojects will not be built or cleaned. The following macro# statements set BUILD_SUB-CONF and CLEAN_SUB-CONF to .build-reqprojects-conf# and .clean-reqprojects-conf unless SUB has the value 'no'SUB_no=NOSUBPROJECTS=${SUB_${SUB}}BUILD_SUBPROJECTS_=.build-subprojectsBUILD_SUBPROJECTS_NO=BUILD_SUBPROJECTS=${BUILD_SUBPROJECTS_${SUBPROJECTS}}CLEAN_SUBPROJECTS_=.clean-subprojectsCLEAN_SUBPROJECTS_NO=CLEAN_SUBPROJECTS=${CLEAN_SUBPROJECTS_${SUBPROJECTS}}# Project NamePROJECTNAME=testPunctuator# Active ConfigurationDEFAULTCONF=DebugCONF=${DEFAULTCONF}# All ConfigurationsALLCONFS=Debug Release# build.build-impl: .validate-impl@#echo "=> Running $@... Configuration=$(CONF)"${MAKE} -f nbproject/Makefile-${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .build-conf# clean.clean-impl: .validate-impl@#echo "=> Running $@... Configuration=$(CONF)"${MAKE} -f nbproject/Makefile-${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .clean-conf# clobber.clobber-impl:@#echo "=> Running $@..."for CONF in ${ALLCONFS}; do ${MAKE} -f nbproject/Makefile-$${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .clean-conf; done# all.all-impl:@#echo "=> Running $@..."for CONF in ${ALLCONFS}; do ${MAKE} -f nbproject/Makefile-$${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .build-conf; done# configuration validation.validate-impl:@if [ ! -f nbproject/Makefile-${CONF}.mk ]; then echo ""; echo "Error: can not find the makefile for configuration '${CONF}' in project ${PROJECTNAME}"; echo "See 'make help' for details."; echo "Current directory: " `pwd`; echo ""; fi@if [ ! -f nbproject/Makefile-${CONF}.mk ]; then exit 1; fi# help.help-impl:@echo "This makefile supports the following configurations:"@echo "    ${ALLCONFS}"@echo ""@echo "and the following targets:"@echo "    build  (default target)"@echo "    clean"@echo "    clobber"@echo "    all"@echo "    help"@echo ""@echo "Makefile Usage:"@echo "    make [CONF=] [SUB=no] build"@echo "    make [CONF=&gltCONFIGURATION>] [SUB=no] clean"@echo "    make [SUB=no] clobber"@echo "    make [SUB=no] all"@echo "    make help"@echo ""@echo "Target 'build' will build a specific configuration and, unless 'SUB=no',"@echo "    also build subprojects."@echo "Target 'clean' will clean a specific configuration and, unless 'SUB=no',"@echo "    also clean subprojects."@echo "Target 'clobber' will remove all built files from all configurations and,"@echo "    unless 'SUB=no', also from subprojects."@echo "Target 'all' will will build all configurations and, unless 'SUB=no',"@echo "    also build subprojects."@echo "Target 'help' prints this message."@echo ""

The structure of this file is very uniform. Every command is implementedin the same way (except .help that just echos the help text to thescreen). It always invokes eventually the configuration specific makefile. Commands may operate on all configurations and on sub-projects too(on by default). For example the default .build command (if you justtype 'make') is:

# build.build-impl: .validate-impl@#echo "=> Running $@... Configuration=$(CONF)"${MAKE} -f nbproject/Makefile-${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .build-conf

Let me decipher this line-noise that makes sense only to make-savvy people. The name of the command is .build-impl. It will execute the .validate-impl command, skip the commented out echocommand (if you remove the # it will print the text between the doublequotes) and run 'make' again on the file nbproject/Makefile-${CONF}.mk(CONF is the active configuration, which is Debug in this case, unlessyou specified CONF=Release when you run 'make'). Finally it will executethe .build-conf command that is defined in the Makefile-${CONF}.mk.This command builds all the sub projects (if necessary) and finallybuild the project itself by invoking the C++ compiler and linker.

It sounds complicated and it is complicated. This is the cleanestmake-based system I have seen with good separation of concerns, veryuniform structure and great extensibility. Most make-based build systemsare simply a mess. The nice thing about NetBeans is that it takes careof all the messy parts and lets you work entirely at the IDE level, butstill allows you to extend the build process at the makefile-level ifyou need to do something special.

Let's take a look at the Makefile-Debug.mk file:

## Generated Makefile - do not edit!## Edit the Makefile in the project folder instead (../Makefile). Each target# has a -pre and a -post target defined where you can add customized code.## This makefile implements configuration specific macros and targets.# EnvironmentMKDIR=mkdirCP=cpjCCADMIN=CCadminRANLIB=ranlibCC=gccCCC=g++CXX=g++FC=# Include project Makefileinclude Makefile# Object DirectoryOBJECTDIR=build/Debug/GNU-MacOSX# Object FilesOBJECTFILES= ${OBJECTDIR}/main.o# C Compiler FlagsCFLAGS=# CC Compiler FlagsCCFLAGS=CXXFLAGS=# Fortran Compiler FlagsFFLAGS=# Link Libraries and OptionsLDLIBSOPTIONS=../../hw/utils/dist/Debug/GNU-MacOSX/libutils.a# Build Targets.build-conf: ${BUILD_SUBPROJECTS} dist/Debug/GNU-MacOSX/testpunctuatordist/Debug/GNU-MacOSX/testpunctuator: ${BUILD_SUBPROJECTS}dist/Debug/GNU-MacOSX/testpunctuator: ${OBJECTFILES}${MKDIR} -p dist/Debug/GNU-MacOSX${LINK.cc} -o dist/Debug/GNU-MacOSX/testpunctuator ${OBJECTFILES} ${LDLIBSOPTIONS}${OBJECTDIR}/main.o: main.cpp${MKDIR} -p ${OBJECTDIR}$(COMPILE.cc) -g -I../.. -o ${OBJECTDIR}/main.o main.cpp# Subprojects.build-subprojects:cd ../../hw/utils && ${MAKE}  -f Makefile CONF=Debug# Clean Targets.clean-conf: ${CLEAN_SUBPROJECTS}${RM} -r build/Debug${RM} dist/Debug/GNU-MacOSX/testpunctuator# Subprojects.clean-subprojects:cd ../../hw/utils && ${MAKE}  -f Makefile CONF=Debug clean

This file includes the project's Makefile (which includes theMakefile-impl.mk), defines a bunch of variables that point to differenttools like C and C++ compilers and it also defines the dependencies ofthe project (in this case just the hw/utils sub-project). Note the .build-conf command that I mentioned earlier when I discussed the .build-implcommand from Makefile-impl.mk. So, there is a lot of interplay betweenthe various make files. This is done in the interest of separating fixedlogic like command invocation from very dynamic parts like the filesthat are contained in a project and its dependencies and also providingclear extension points (the.pre and .post commands in the mainMakefile). The bottom line is that most developers don't even need toknow that there is a make-based build system underneath and can juststay at the IDE level. Build administrators can automate the buildprocess using this clean and standard make interface and people withspecial needs can customize the build process very elegantly using theextension points provided by .pre and .post targets, as well as add newtargets.

Project Metadata Files

Make files are okay (I wouldn't say great) for running a build, but theyare not very easy to parse and modify. NetBeans uses two XML files tomaintain the project information and dynamically generates the makefiles from these files.

configurations.xml

This is the file. It contains most of the project information used forgenerating the make files as well as some GUI information such as thelogical folders displayed in the IDE for each project and what filesreside in them. It is a typical XML file. The root element is called configurationDescriptor and it contains a logicalFolder element called "root that contains nested logicalFolder elements for "HeaderFiles", "ResourceFiles", "SourceFiles" and "ExternalFiles". Each logicalFolder element may contain itemPath elements for the files in this folder. Here is the logical folders of the testPunctuator project:

main.cppMakefile

The configurations.xml file also contains elements for the encoding themake file for the project (in case you want to change its name for somereason):

  UTF-8Makefile

Then comes the all important "confs" elements that contains "conf"elements for each configuration. Each "conf" elements contains a"toolset" element and a "compileType" elements that contains varioustools. Each tool has its own set of elements and attribute thattranslate directly to make file tool settings:

GNU|GNU4../..1GNU|GNU455../..51

There is one "conf" element for each configuration (in this case Debug and Release).

project.xml

The project.xml file holds additional information like the project type,project dependencies and the file extensions of different file types.I'm not sure why this information should go in a separate file, butthat's how it is. Here is the project.xml file of the hello_worldapplication project itself:

  org.netbeans.modules.cnd.makeprojecthello_world0../../hw/hello../../hw/utils../../hw/worldcpphpp

Project Group

In addition to the individual project files NetBeans support a higherlevel of organization called a project group. A project group is simply acollection of projects that can be loaded together into the NetBeansIDE. The hello world system is also a project group that contains manyprojects. The ibs can generate such a project group on behalf of theuser and update it automatically when new projects are added or removed.

NetBeans keeps a lot of information in the user's home directory in ahidden directory called ".netbeans". The project groups are stored inthe following directory:

~/.netbeans/6.7/config/Preferences/org/netbeans/modules/projectui/groups

Note the 6.7 version number following the .netbeans directory. You mayhave multiple versions of NetBeans installed on your machine and theirpreferences are stored separately. Each project group has a file underthe projectui sub-directory called .properties.There are three kinds of projects groups: free group, master +dependencies and folder group (all projects under a root directory). Thehello_world project group is a folder group. Here is thehello_world.properties file:

name=hello_worldkind=directorypath=file\:/Users/gsayfan/Documents/Invisible.Build.System/src

Pretty simple, really. In addition there is another important file called:

~/.netbeans/6.7/config/Preferences/org/netbeans/modules/projectui/groups.properties.

This file determines the active project group if there are multipleproject groups. Its format is very simple too and to make a group activeyou just need to have this line in the file:

active=

In case of the hello_world project group it is:

active=hello_world

The NetBeans Helper

The NetBeans Helper class is responsible for implementing all the codethat is NetBeans-specific. The generic build_system_generator.py scriptis using this helper to generate all the NetBeans project files (insidethe nbproject directory) for each project and a project group thatincludes all the generated project. Let's take a closer look at thisclass. The first thing it does is import some useful system modules andthen import the BaseHelper and Template classes from thebuild_system_generator module (as well as the 'title' function fordebugging purposes):

#!/usr/bin/env pythonimport os, sys, stringfrom pprint import pprint as ppsys.path.insert(0, os.path.join(os.path.abspath(os.path.dirname(__file__)), '../'))from build_system_generator import (BaseHelper,Template,title)

Then the Helperclass is defined. This is the class that the buildsystem generator module is using to customize the build systemgeneration for each specific target (NetBeans 6.x in this case). The __init__() method accepts the templates_dir,which is the path to the root of the templates directory used togenerate all the build files. It also initializes the separator ('/')and line separator ('\n') to Unix values to make the generated files fitwell in their intended environment. The skip_dir is used to tellthe recursive drill-down code that looks for projects insub-directories to ignore directories called 'nbproject' (which is thespecial sub-directory used by NetBeans to store the build files). Theother methods this class implements are: get_templates(), prepare_substitution_dict(), and generate_workspace_files().

class Helper(BaseHelper):"""NetBeans6 helper"""def __init__(self, templates_dir):BaseHelper.__init__(self, templates_dir)self.sep = '/'self.linesep = '\n'self.skip_dir = 'nbproject'def get_templates(self, template_type):...def prepare_substitution_dict(self, ...):...def generate_workspace_files(self, name, root_path, projects):...

get_templates()

The get_templates()method is pretty simple. For each build filethere is a corresponding template file. These template files are justskeleton of real build files, with some place holders. You will see allthe template files soon enough. The get_templates() method justiterates over all the template files (located in the nbproject) and addsa template for the Makefile in the project directory itself. For eachsuch build a file a Template object is generated. Finally the list of Template objects is returned.

  def get_templates(self, template_type):result = [Template(os.path.join(self.templates_dir,template_type,'Makefile'),'Makefile',template_type)]nb_project = os.path.join(self.templates_dir, template_type, 'nbproject')assert os.path.isdir(nb_project)for f in os.listdir(nb_project):project_file_template = os.path.join(nb_project, f)if not os.path.isfile(project_file_template):continuefilename = os.path.join(nb_project, f)relative_path = '/'.join(['nbproject', f])result.append(Template(filename, relative_path, template_type))return result

prepare_substitution_dict()

This method is the heart of NetBeans6_Helper class. It isresponsible for creating a substitution dictionary that contains all thevalues to be substituted into the templates of each build file. This isnot so trivial because some place holders are supposed to be replacedby dynamic content that is generated on the fly. In addition, as you sawearlier NetBeans has quite a few build files. The prepare_substitution_dict()method has several nested function to assist in prepare thesubstitution dictionary for each one of them. The nested functions are:

• prepare_makefile() for generating the Makefile-Debug.mk and Makefile-Release.mk files
• prepare_configurations_xml() for generating configurations.xml
• prepare_project_properties() for generating project.properties
• prepare_project_xml() for generating project.xml

The substitution dict for the project's main Makefile is empty becauseit is a generic file that doesn't have any place holder and thesubstitution dict for the Makefile-impl.mk file contains only the nameof the project so no helper function is necessary. Here is the code ofthe method (without the nested functions). It accepts a long list ofarguments that the various nested functions use to generate the propervalues. The operating system and the dynamic library extension are alsodetermined here. This method is called multiple times with differenttemplate names (each template_name corresponds to a build file) and prepare_substitution_dict() calls the proper nested function or generates the dict directly (for Makefile and MakeFile-Impl.mk).

def prepare_substitution_dict(self,project_name,project_type,project_file_template,project_dir,libs_dir,dependencies,source_files,header_files,platform):if platform.startswith('darwin'):operating_system = 'MacOSX'ext = 'dylib'elif platform.startswith('linux'):operating_system = 'Linux'ext = 'so'temaplate_name = os.path.basename(project_file_template)if temaplate_name ==  'Makefile':return {}if temaplate_name ==  'Makefile-Debug.mk':return prepare_makefile('Debug', operating_system)if temaplate_name ==  'Makefile-Release.mk':return prepare_makefile('Release', operating_system)if temaplate_name == 'Makefile-impl.mk':return dict(Name=os.path.basename(project_dir))if temaplate_name == 'configurations.xml':return prepare_configurations_xml(operating_system)if temaplate_name == 'project.properties':return prepare_project_properties()if temaplate_name == 'project.xml':return prepare_project_xml(dependencies)assert False, 'Invalid project file template: ' + temaplate_namereturn {}

Now, let's examine one of nested functions. I chose the prepare_makefile()function because it is not trivial. The keys in its substitutiondictionary are: 'ObjectFiles', 'CompileFiles', 'LinkCommand','LDLIBSOPTIONS', 'BuildSubprojects', 'CleanSubprojects','OperatingSystem' and 'DynamicLibExtension'. Some of these are simplestrings like 'OperatingSystem' and 'DynamicLibExtension'. Others aremuch more complicated like 'CompileFiles', which is a list of compilecommands where each command itself requires a template with substitutionvalues such as 'File', 'CompileFlag', 'Platform' and 'FPIC'. The link command depends on the project type and ldliboptions depends on the platform. Here is the code:

    def prepare_makefile(conf, operating_system):compile_flag = '-g' if conf == 'Debug' else '-O2'd = dict(Name=project_name)object_file_template = '${OBJECTDIR}/%s.o \\\n'object_files = ''for f in source_files:f = os.path.splitext(os.path.basename(f))[0]object_files += object_file_template % f# Flag for dynamic librariesfpic = '-fPIC  ' if project_type == 'dynamic_lib' else ''# Get rid of last forward slashif len(object_files) > 2:object_files = object_files[:-3]d['ObjectFiles'] = object_filescompile_file_template = '$${OBJECTDIR}/${File}.o: ${File}.cpp \n' + '\t$${MKDIR} -p $${OBJECTDIR}\n' + '\t$$(COMPILE.cc) ${CompileFlag} -I../.. ${FPIC}-o $${OBJECTDIR}/${File}.o ${File}.cpp\n\n't = string.Template(compile_file_template)compile_files = ''for f in source_files:f = os.path.splitext(os.path.basename(f))[0]text = t.substitute(dict(File=f,CompileFlag=compile_flag,Platform=platform,FPIC=fpic))compile_files += text# Get rid of the last two \n\n.compile_files = compile_files[:-2]d['CompileFiles'] = compile_fileslink_command = ''if project_type == 'dynamic_lib':if platform.startswith('darwin'):link_command = '${LINK.cc} -dynamiclib -install_name lib%s.dylib' % project_nameelse:assert platform.startswith('linux')link_command = '${LINK.c} -shared'd['LinkCommand'] = link_commandldlibsoptions = ''if dependencies != []:ldliboption_template = '../../hw/%s/dist/%s/GNU-%s/lib%s.a'ldlibsoptions = ' '.join([ldliboption_template % (dep.name, conf, operating_system, dep.name)for dep in dependencies])if operating_system == 'Linux':ldlibsoptions += ' -ldl'd['LDLIBSOPTIONS'] = ldlibsoptionsbuild_subproject_template = '\tcd ../../hw/%s && ${MAKE}  -f Makefile CONF=%s'clean_subproject_template = build_subproject_template + ' clean'build_list = [build_subproject_template % (dep.name, conf) for dep in dependencies]clean_list = [clean_subproject_template % (dep.name, conf) for dep in dependencies]d['BuildSubprojects'] = '\n'.join(build_list)d['CleanSubprojects'] = '\n'.join(clean_list)d['OperatingSystem'] = operating_systemd['DynamicLibExtension'] = extreturn d

Note, that there are better ways to accomplish this task. There areseveral third-party template languages like Genshi, Mako, Tempita andJinja. These template engines can handle the nested templates that prepare_makefile()generates manually in a much more natural way. The code could have beenmuch shorter and concise. I made a deliberate decision to use onlystandard Python libraries in the interest of keeping the scope of thisproject limited. Choosing a particular template language/engine wouldhave made the code shorter, but required the reader to understand anadditional language and might antagonize fans of other templatelanguages.

The other prepare_XXX() nested functions are all very similar to make_makefile() although some of them generate XML files and another one generate a properties file (INI file like).

generate_workspace_files()

This method is responsible for generating the project groups in the user account. The reason the method is called generate_workspace_files() is that the method is defined in the generic Helper base class and NetBeans6_Helperis just overriding it. So, the NetBeans-specific term "Project Group"is not used here. The code itself is pretty simple. It either creates orupdates the proper .properties files that dictate the contents of theproject groups as explained earlier:

  def generate_workspace_files(self, name, root_path, projects):"""Generate a NetBeans project group for all the generated projects"""base_path = '~/.netbeans/6.7/config/Preferences/org/netbeans/modules/projectui'base_path = os.path.expanduser(base_path)if not os.path.exists(base_path):os.makedirs(base_path)# Create a project groupgroups_path = os.path.join(base_path, 'groups')if not os.path.exists(groups_path):os.makedirs(groups_path)text = """name=%skind=directorypath=file\:%s"""group_filename = os.path.join(groups_path, name + '.properties')open(group_filename, 'w').write(text % (name, root_path))# Make it the active projecttext = 'active=' + nameopen(os.path.join(base_path, 'groups.properties'), 'w').write(text)

The NetBeans Project Templates

The substitution dictionaries are very important of course, but theycan't do much by themselves. Each build file is generated bysubstituting the values from the proper dictionary into the propertemplate file.

As you recall NetBeans can build three types of projects: staticlibrary, dynamic library and a program. for each one of them there aretemplates of all the build files. A few templates are the same for someor all project types, so an identical copy is kept for each one. Thetemplates are organized in the following file system structure:

project_templatesNetBeans_6dynamic_libMakefilenbprojectconfigurations.xmlMakefile-Debug.mkMakefile-Impl.mkMakefile-Release.mkproject.propertiesproject.xmlprogramMakefilenbproject...static_libMakefilenbproject...

This regular structure mimics the structure of the build files inside aproject directory and allows the generic part of ibs to apply thesubstitution dicts to the templates blindly and end up with the correctbuild file in the correct place. Note, the project.properties file thatwasn't mentioned earlier. This is an empty file that doesn't seem tohave a role in C++ projects, but I keep it there to be consistent withNetBeans.

To create the template files I simply took the various NetBeans buildfiles and replaced anything that was project-specific (like the sourcefiles or list of dependencies) with a place holder.Let's examine acouple of template files. Here is the main part of the Makefile-Debug.mkof the 'program' project type:

# Link Libraries and OptionsLDLIBSOPTIONS=${LDLIBSOPTIONS}# Build Targets.build-conf: $${BUILD_SUBPROJECTS} dist/Debug/GNU-${OperatingSystem}/${name}dist/Debug/GNU-${OperatingSystem}/${name}: $${BUILD_SUBPROJECTS}dist/Debug/GNU-${OperatingSystem}/${name}: $${OBJECTFILES}$${MKDIR} -p dist/Debug/GNU-${OperatingSystem}$${LINK.cc} -o dist/Debug/GNU-${OperatingSystem}/${name} $${OBJECTFILES} $${LDLIBSOPTIONS}${CompileFiles}# Subprojects.build-subprojects:${BuildSubprojects}# Clean Targets.clean-conf: $${CLEAN_SUBPROJECTS}$${RM} -r build/Debug$${RM} dist/Debug/GNU-${OperatingSystem}/${name}# Subprojects.clean-subprojects:${CleanSubprojects}

The placeholder are expressions of the form ${Place holder}. This is the format used by the string.Templateclass. Unfortunately, this convention is used by make files a lot toofor environment variable, defined symbols and make variables. So, when a$ sign is part of the make file it is escaped by additional $ sign. For example, $${MKDIR} will not be treated as a place holder by the ibs.

Here is a simpler template of the project.xml file. It is just a bunchof XML with two place holders for the name of the project and itsdependencies:

org.netbeans.modules.cnd.makeproject${Name}0${MakeDepProjects}

Testing the NetBeans Generated Build System

Bob was pleased with ibs and he decided to put it to the test. His planto make sure it works on Mac OS X and on Linux (Kubuntu 9.04). For eachtarget OS Bob generated all the build files using ibs and then built allthe projects both from the command line using make and from theNetBeans IDE itself. Then he proceeded to run various tests andprograms.

Generate all the build files

Generating the build files is as simple as launching the build systemgenerator. The program displays some simple progress information as itgoes through the different stages.

{root dir}/ibs >  py build_system_generator.py--------------------generate_build_files------------------------------------------_populate_project_list--------------------------test--------dlls--------apps------hw-------------------generate_projects---------------------------------------_generate_project world------------------------------------------------_generate_project testWorld------------------------------------------------_generate_project utils--------------------------------------------_generate_project hello-------------------------------------------------_generate_project punctuator-------------------------------------------------_generate_project utils-----------------------------------------------------_generate_project testPunctuator-----------------------------------------------------------_generate_project hello_world------------------------------------------------------_generate_project testHello---------------------------------------save_projects-------------------------------------generate_workspace_files------------------------

Bob did a quick sanity check to verify that the nbproject directory was indeed created for each project under the src directory:

{root dir} > find src -name nbprojectsrc/apps/hello_world/nbprojectsrc/dlls/punctuator/nbprojectsrc/hw/hello/nbprojectsrc/hw/utils/nbprojectsrc/hw/world/nbprojectsrc/test/testHello/nbprojectsrc/test/testPunctuator/nbprojectsrc/test/testWorld/nbproject

At that point Isaac (the sage) heard the good news and came into theroom. He wanted to personally supervise on the testing of ibs, whichwill soon be responsible for building the entire "Hello World -Enterprise Edition" system.

The next stage was to actually build the system using the ibs-generatedbuild files. For starters Bob fired up NetBeans on Mac OS X, built andran the hello_world application. This caused all its dependencies to bebuilt and finally the application ran in its little terminal window andindeed printed the vaunted "hello, world!" message (see Figure 2).

Figure 2

Isaac was duly impressed, but not fully convinced yet. He asked Bob howibs can handle automated tests and running outside of the NetBeans IDE.Bob was more than happy to comply and demonstrated how the test_worldprogram can be built using the standard 'make' command from a terminalwindow and then executed (see Figure 3).

Figure 3

Isaac commended Bob on a job well done, but Bob wasn't done. He knewthat Isaac was an old Unix hand and he proceeded to demonstrate theibs-generated build files on Kubuntu 9.04 (in a VM). First he built thelibPunctuator project in the NetBeans IDe and then the testPunctuatorproject (see Figure 4).

Figure 4

Bob was careful to copy the "libpunctuator.so" shared library to thedirectory of the testpunctuator program because the test always tries toload the shared library from the current working directory. They bothnoted with interest that on Kubuntu 9.04 programs run in an externalterminal window (see Figure 5) as opposed to the internal NetBeanswindow on the Mac OS X.

Figure 5

Then, Bob demonstrated building from Kubuntu's terminal (see Figure 6).

Figure 6

Isaac felt his concerns melting away. He was now convinced that ibs isthe way to go to make "Hello World - Enterprise Edition" the best helloworld application on Microsoft Windows too on the way to [Hello] WorldDomination!

Conclusion

In this article you saw ibs in action, generating a full fledgedNetBeans build system for a non-trivial system that involves multipleprojects, static libraries, shared libraries, applications and testprograms. ibs handled well multiple target operating systems (Mac OS Xand Kubuntu 9.04) and allowed building and testing from the NetBeans IDEor externally from a terminal window. Bob demonstrated ibs successfullyto Isaac his manager and in the next episode, Bob will try to make ibsbuild the "Hello World - Enterprise Edition" system on the Windows OS.