| Version Control Strategy for Windows CE Platform Environments |
| Written by Henrik Viklund | |
|
Platform Builder is one of those environments that really resist being put under version control. Especially if you’re new to Platform Builder, it can be a very frustrating and time consuming experience to put the environment under version control. It’s not just the cheer number of files that a Windows CE image is built from that can be hard to cope with, but also the fact that some of the Platform Builder tools don’t cope well with some version control systems. What this article aims for is not to describe some sort of divine methodology that will work for all situations, because I don't beleave such methodologies exist. Instead, I'll try to give you a starting point; I’ll describe what I consider the minimum amount of ceremony needed to effectively version control a Platform Builder environment –the least amount of complexity that still floats the boat, so to speak. Also, I’ll cover the most usual workarounds that are needed to get a version controlled Platform Builder Environment up and running.
This article targets Windows CE 5, but I believe that the general principles discussed here apply to most versions of the Windows CE source tree. Some DefinitionsGenerally I’ve tried to use the definitions from this wikipeda article about revision control. I use the term “version control” throughout the article, because I think it is a bit more general than “source control”, and a bit more self-explaining than “revision control”. When I talk about Platform Builder in a version control context, I am refering to the source tree, “…\WINCE500\...”, that Platform Builder uses as base for creating an OS image, not the IDE itself. Also, as you may notice, I use “VC” as an acronym for “Version Control” every now and then. Why version control?I want to know what’s being shipped, and I want to endorse productivity through concurrency. I want to know what’s being shipped in a certain version of the product because it’s usually my butt that ends up in the frying pan when the bug reports starts to drop in. I want productivity and concurrency because I hate having my butt in the frying pan longer than necessary. With proper verison control, I have a better chanse of succeeding with my projects since it:
A few words about choosing a version control systemSo before actually explaining how I would go about version controlling Windows CE platform, just let me share with you what I find important in a good version control system. Basically, my two most important criteria for a version control system are:
The ability to branch and merge efficiently is crucial for handling updates and releases the way I want. Atomic commits allows developers to construct and commit changes to multiple files as one logical change, rather than treating each file as a separate commit. With atomic commits, you’ll be able to track exactly what files where affected by a specific bugfix, feature request etc. I find this very useful. With these two criteria I effectively disqualify for example Microsoft Source Safe, (and possibly also CVS) as a version control system suitable for Platform Builder. While it implements rudimentary branch and merge support, it is just not good enough for the kind of version control I think is needed for PB. Source Safe also lacks atomic commits, and with the massive amount of files, it’s next to impossible to figure out exactly what files where changed to fix that bug, or implement that feature. Sorry Microsoft, it’s just not good enough. However, there are myriads of other commercial version control systems out there that will do the job just fine. If you don’t feel like spending megabucks on a commercial alternative, Subversion and TortoiseSVN is a great open source version control system and UI combo that is worth checking out. The big pictureThe CE build tree is huge. A full install puts of well over 100000 files, and takes nearly 13 Gb of space on your hard drive. It never stops amazing me how huge this tool is. But, from a VC point of view, the Platform Builder tree can be divided into two parts:
Basically, I define “Their Content” as made up of all the files that where put into the tree at install. It is considered bad practice to change any files provided by Microsoft. Rather than change files directly, you are advised to clone (copy) the parts of the tree that needs to be modified to a different location and modify the cloned files instead. “Your Content” is *drum roll* the parts of the tree that you add and make changes to. Usually, you’ll need to adapt a BSP, and you’re using that BSP in a platform project. The BSP and any platform projects are “Your Content”. Also, everything else you clone or create falls under this category. Why I like to divide the build tree into “Theirs” and “Yours” is that although it's part of the same tree, they differ in the way they evolve over time, and who evolves them, and that has implications on how to version control them. We’ll get to that soon. Just to sort out where “Your content” usually ends up, physically, BSP:s are located under ”WINCE500\PLATFORM\<some bsp>”. A project on the other hand may or may not be located under the root build tree (“WINCE500” or whatever version or CE you’re working with). The default location for a project is under ”…\WINCE500\PBWorkspaces\<a project>”, but you can put your projects pretty much anywhere you like. Usually I recommend sticking to the standard locations if you don’t have a good reason not to. So, to summarize, files provided by Microsoft equals stuff that you as a developer should not change. Stuff that you need to change you either clone from Microsoft’s files, create from scratch, or get from 3:rd party vendors. One typical use-case for cloning files is when modifying or creating a BSP for your hardware. Usually, you use an existing BSP as base, and modify it to fit your needs. Rather than modify the files directly (bad practice, remember?), you clone the BSP to a new BSP sub-tree and modify those files instead, keeping the original intact.
What to Put Under Version control?Well, the vague answer is that it depends on your specific needs. Some advocate only “Your Content”, like the BSP you’re developing, or the platform project you’re working on needs to be put under version control. The motivation usually is that since you should not change the contents of the original tree, this will remain constant over time, and you really only need to put the parts that you change or add under version control. The static part you can recreate by just reinstalling Platform Builder, should it be needed. I, on the other hand, advocate that you should put everything under version control. Why? Well, this might come as a surprise to you, but software is known to contain bugs (!). Since Windows CE is no exception, Microsoft releases Quick Fix Engineering Updates, QFE:s, every now and then. When installed, a QFE updates and/or adds files to the build tree. In other words, the contents that is thought of as constant over time, is actually pretty volatile. Now, if, or rather when, you get a tough to figure out bug report for a device that’s been on the market for a while it’s often a huge advantage to be able to recreate the exact build environment used for that particular image. Since all the compilers, linkers and other tools used to build the image resides in the build tree, version controlling the whole tree, you can just check out that version and voila, you’ve recreated that build environment. You’ll be able to look at the source code as it looked at the time the image was created, create some hacks to reproduce the bug, pin down a fix etc. Without the support from the version control system, It is just so much more labour intense and error prone to recreate a build environment, taking into account that you need to keep track of what QFE:s and rollups had been applied at the time of the build etc. If you put the whole tree under version control you’ll not only ease product archaeology expeditions like the one mentioned above, you’re also making it a lot easier to maintain up-to-date, identical build environments for all team members. If their working copy is up to date, they’re good to go. Sure, putting the whole tree under VC will take considerably more storage space and bandwidth, but since you get storage and bandwidth almost for free these days, it’s a really cheap insurance. What not to put under version control?Huh? I just said everything should go under version control, didn’t I? Yes, I did, and I stick to that –with the following exception; when sysgen:ing the image, there’s whole bunch of intermediate files created at various places throughout the build tree. These, you generally don’t want to put under version control (unless you can come up with a very good reason to do so of course!). Cleaning upA BSP: As I said earlier the BSP resides under “…\WINCE500\PLATFORM\<some bsp>”. The following files are intermediate files or otherwise unnecessary to version control:
A Project: In the project folder, “%_PROJECTROOT%”, the following files are intermediate files or otherwise unnecessary to version control:
Importing the treeThe easiest way of putting the correct files under version control is to import a fresh PB install in which nothing has been built yet. If you’ve built projects already, you should either scrap the tree and start over (i.e. reinstall), or you can try to filter out or delete the intermediate files created during a build as described before. After you’ve checked in the "squeeky clean" tree, you naturally go right to Microsoft’s download site and download all relevant rollups and QFE:s. I’d probably apply the rollups and QFE:s, one by one, checking in the changes after each install, meticulously logging each installed package. However, if you’re not really interested in this level of record keeping you can apply all the QFE’s and rollups to this date before actually importing the tree. I’ll leave it up to you to decide. Use a vendor branchI prefer to keep the “clean” build tree, “Their content”, in what is known as a vendor branch. A vendor branch is a CM-pattern used to handle 3:rd party resources that your project is closely related to, or perhaps dependent upon. Generally, the needs of your project will dictate that you stay as up-to-date as possible with the resources provided by that 3:rd party vendor without jeopardising the stability or feature set of your own project. The vendor branch is an elegant solution to this problem. When using a vendor branch, updates from the 3:rd party vendor (usually dubbed “vendor drops”) are absorbed into a branch separate from your project’s branch. The vendor branch is then merged into the project branch when the project is ready to absorb the vendor drop. Once merged into the project branch, the code can be patched and altered to your liking in the project branch, without polluting the original vendor drop with project specific mutations. In our case, Microsoft is the 3:rd party vendor, and a QFE or a rollup update will be a vendor drop. It might seem like overkill to use this approach, especially since altering any file provided by Microsoft is considered bad practice. Rather than altering the file directly, you should always clone the file and alter the cloned version leaving the original as is. While cloning is doable in almost all cases, I have found a couple of rare exceptions where it is not possible, or at least next to impossible to tie everything together without having to alter Microsoft’s version of a file. One example is if you need to modify certain parts of GWES, such as “oomui.lib”. This modification involves altering ”winceos.bat”, which is not easily cloned and thus you need to alter it directly. So, because you can’t guarantee that it is always possible (or that you're skilled enough) to duplicate the code in every case, I advocate the use of vendor branches, which separates the changes made to the tree by Microsoft, and the changes made specifically for your project. A decent version control system will detect any alterations made in the project branch since last vendor branch merge, so the possibility to overwrite any project specific changes done to the vendor specific contents by mistake is practically eliminated. Had the vendor drop, (i.e a QFE) been installed directly onto a working copy of the project branch and then checked in, conflicting changes might have been overwritten by the QFE install. Not good.
The vendor branch pattern can also be used to manage other 3:rd party code such as a 3:rd party BSP. Normally, each 3:rd party package should have its own vendor branch, so you can control the updates independently from one another. Creating a Working Working CopyOK, so now you’ve got the tree under version control. Now it’s time to set up a working copy that will actually work! File permissionsAs you probably know, it is common for version control systems to manipulate the read-only attributes of the files under version control. Unfortunately some tools used during a sysgen don’t cope very well with read-only files. For example, it can be that the build system needs to replace a file with a localized version of it. This mean that if you have a version control system that sets read-only attributes on files, you need to handle this and remove the read only attributes for these files. If you don’t do that, the build system will complain with some not-that-obvious errors when trying to sysgen your working copy. Basically you have two choices. Either you setup your VC system to place these files as read/write in the working copy, or you need to make them read/write in some other way. The problematic filesSo what files needs to be writable then? Usually, read/write-enabling “*.res” in the directories and subdirectories listed below will get you off the hook. Under special circumstances "*.lib" and "*.pdb" contained in theese directories may also need to be writable, but I advice you to start only with "*res":
%_WINCEROOT%\public\common\oak\lib\%_TGTCPU%\%WINCEDEBUG%\
Actually, not all files may need to be writable, but I'll leave it as an excersise for you to figure out exactly what files you need to modify the permissions for.
Brute Force approachOK. An ugly problem calls for an ugly solution. If you can’t get your VCS to relax the read-only attributes for the files mentioned above in a controlled manner, you might be able to use a batch file to brute-force the read/write permission on the files. Preferably you put it as a “pre-sysgen” custom build action. It will run through the offending files and remove any read-only attributes on them. If your VCS is smart and checks these permissions before updating the working copy (like for example Perforce), you also need a way of reverting to the read only attributes or the VCS will complain (hint: use a "post-sysgen" script). An example batch file may look something like this:
attrib -R %_WINCEROOT%\public\common\oak\lib\%_TGTCPU%\%WINCEDEBUG%\*.pdb As you’ve probably figured out, using a batch file to modify the permissions is a quick n’ dirty solution, so you should try to avoid it and let the VCS handle it if possible.
Other ConsiderationsApart from being a general pain to version controll, there are a number of other things that you should know about when setting up a version controlled build environment for Platform Builder. Install as little as possibleIf you don't plan to develop for all architectures, don't install them. For each additional architecture, a large amount of files will be copied into the build tree, most of which are binary files (libs) that generally are costly to version control. By keeping the tree as slim as possible you not only save disk space, you cut down on time spent checking out a new tree or switching branches, you also only have to worry about keeping the current architecture set up to date with QFE:s and rollups. If you need another architecture, you can allways install and import it tothe VCS later. Catalog DatabaseA classic problem you’re bound to run into is when updating or importing CEC files into the catalog. PB actually uses an Access database to store this information. Before you can get anything into that database you need to check it out. The “.mdb” database is located under “…\WINCE500\PUBLIC\COMMON\OAK\CATALOG\DATABASE\pbdb.mdb”. SDK generationAnother caveat is with SDK generation. While most of PB’s tools rely on system variables to find it’s way round the source tree, The SDK wizard is practically system variable agnostic. For example, all extra files you specify to go into the SDK are “absolute path”. If your project members have their projects checked out to different paths on their respective machines, it will break the SDK wizard. Also, the SDK wizard has a habit of failing silently, so be sure to check out the SDK files before editing the SDK. Evaluation versionPutting a build tree from an evaluation version of Platform Builder under version control might seem like a smart thing to do, giving you the ability to develop part of your design while your Platform Builder Full Version order is snail-pacing its way through your company’s internal paperwork. As smart as it seems, it might create some pretty nasty problems later on. The evaluation version is “time-bombed” and expires after a couple of months. Since all tools needed to build a platform reside inside the build tree, you’ve actally put the “time-bombed” evaluation versions under version control. When checking out the working copy, this will overwrite the commercial versions of the tools. So, remember to import a fresh tree once you get the full version of the tool, or otherwise the build system will mysteriously stop working after a couple of months, in spite of having the full licensed version of Plaform Builder.
As usual, I highly appriciate any comments on this article. Especially bad splelling or if you have found errors in this article, or of you think you have some relevant feedback that wil make this article better.
|