Highly recommended.
Continuous Delivery, by Jez Humble and David Farley, is subtitled : `Reliable Software Releases through Build, Test and Deployment Automation'. And indeed, all of this can and should be automated!
The authors describe everything you should know about why and how to automate software releases, from the commit to the deployment on live production servers. I already knew about Continuous Integration when I started reading this book, so I didn't learn much about this topic. But if you are not already familiar with this concept, just stop now, get the book, and read the first three chapters! That's the absolute minimum you should do! And that's enough to highly recommend it.
The book is geared toward building applications that get released on the company's servers, rather than shipped to customers. This is very different from the project I'm currently working on, for which we essentially deliver libraries that customers will include in their own applications. But most of the content of the book is still completely relevant: it doesn't really matter whether you deploy or ship: you should still release often!
Chapter 11 is entirely devoted to 'Managing Infrastructure and Environments'... Did you know that you can store your test servers' configuration under source control? That you can turn bare metal into a running server configured for your application completely automatically, and thus reliably? That is something I want to implement!
Chapter 14 deals with 'Advanced Version Control'. As the authors put it, "poor version control practices are one of the most common barriers to fast, low-risk releases". VCS technologies are discussed and compared (centralized, distributed, stream-based), as well as branching patterns (develop on mainline, branch for release, branch by team, and branch by feature). Very important, and very interesting.
This book is definitely a must-read for anyone responsible for delivering software!...
Showing posts with label SVN. Show all posts
Showing posts with label SVN. Show all posts
Sunday, November 25, 2012
Thursday, September 13, 2012
Suggestions for Jenkins on multi-platform projects
Our team uses Jenkins as our Continuous Integration tool. I would like in this post to describe our usage, and suggest a few ideas that could improve this great tool. But first let me explain what we are building...
The product that we are building is a Mathematical Programming engine. The most basic usage is that you feed it with the mathematical formulation of the business problem that you want to solve, and ask for the best possible solution to this problem. The program then cranks up all the CPUs/cores it can find on your machine and returns with an answer after a few tens of a second, or a few hours (some problems are REALLY complicated).
The core of the engine is a library built from 500.000 lines of C code. In addition to this library, we have APIs in half a dozen other languages (C++, Java, Python, etc), and connectors for several third party applications. No less than 14 platforms (including Windows, Linux on various CPU types, MacOs, AIX, HP-UX, etc.) are supported.
We are therefore very glad to use continuous builds: you really don't want to discover a possible compiler bug, or a non-determinism in the code on some exotic platform just before the release!
On our master branch (the one that receives most of the commits from the developers) we use two job families.
The first one builds the software in Debug mode and runs a fully comprehensive suite of tests. We have around 20 such jobs for all the platform/compiler/settings combinations we support. Run times vary widely: some of the jobs are done in 1h30, while others need almost 7 hours.
The second family (the 'distrib' jobs) builds the Release versions of the product. There is one job per platform. Each job builds all the components for this platform (e.g. on Windows32, we support both Visual Studio 2008 and 2010), packages them into some releasable form (could be a Zip, a TarZ or an installer) and tests the basic functionality of the software (e.g. the distributed samples). For those jobs, the run times vary even more: from 30 minutes to 10 hours, depending on the platform.
This setup has been in place for some time now. It works, and it's extremely useful!
Although Jenkins is a great tool, it doesn't yet have all the features I'd like. So here are a few ideas, just in case the developers would not have already enough...
We sometimes have jobs that stop running (no new run is triggered, or no available nodes). This is of course not intended, and it would be nice to be able to detect those easily. I suppose that adding a 'Last build' column to the list view, that would display the time since the job entered its current state, would be nice. Something like 'Ended 8.6 hr' or 'Queued 1.3 hr' or 'Started 12 min'...
Then I'd know that if the code changed 3 hours ago, I shouldn't see any number larger than 3 hours...
We have many jobs running, typically 20 to 30 simultaneously. And some builds last for several hours. It happens that tests hang, or are abnormally slow. These situations should be detected as soon as possible for investigation.
Unfortunately, the 'Build History' list is not very helpful, for two reasons. It has too few jobs for us: with 50 builds, only the last 5 hours are covered, which is less than the duration of many of our builds. But then if this limit was increased, we'd probably need a list of 200 or so jobs, which would not be easy to handle.
I would thus suggest to allow filtering on the 'building' status. When this flag would be set, the 'Build History' would only display the jobs that are currently being built.
I often need to check if a given revision of the source has been built by a given job, or what is the latest revision that is good on a set of jobs. For example, I may want to merge this revision to some 'stable' branch for other teams to use.
I think that a grid view with the following attributes would be very useful for this: each line is a commit id or SVN revision, each column is a job, each cell is blue, red or gray (or even empty if this revision has not yet been part of a run of the job, or the run is not finished yet).
Do you think these would be useful additions?
What we build
The product that we are building is a Mathematical Programming engine. The most basic usage is that you feed it with the mathematical formulation of the business problem that you want to solve, and ask for the best possible solution to this problem. The program then cranks up all the CPUs/cores it can find on your machine and returns with an answer after a few tens of a second, or a few hours (some problems are REALLY complicated).
The core of the engine is a library built from 500.000 lines of C code. In addition to this library, we have APIs in half a dozen other languages (C++, Java, Python, etc), and connectors for several third party applications. No less than 14 platforms (including Windows, Linux on various CPU types, MacOs, AIX, HP-UX, etc.) are supported.
We are therefore very glad to use continuous builds: you really don't want to discover a possible compiler bug, or a non-determinism in the code on some exotic platform just before the release!
Our current setup
On our master branch (the one that receives most of the commits from the developers) we use two job families.
The first one builds the software in Debug mode and runs a fully comprehensive suite of tests. We have around 20 such jobs for all the platform/compiler/settings combinations we support. Run times vary widely: some of the jobs are done in 1h30, while others need almost 7 hours.
The second family (the 'distrib' jobs) builds the Release versions of the product. There is one job per platform. Each job builds all the components for this platform (e.g. on Windows32, we support both Visual Studio 2008 and 2010), packages them into some releasable form (could be a Zip, a TarZ or an installer) and tests the basic functionality of the software (e.g. the distributed samples). For those jobs, the run times vary even more: from 30 minutes to 10 hours, depending on the platform.
This setup has been in place for some time now. It works, and it's extremely useful!
Whishes
Although Jenkins is a great tool, it doesn't yet have all the features I'd like. So here are a few ideas, just in case the developers would not have already enough...
Detect stale jobs
We sometimes have jobs that stop running (no new run is triggered, or no available nodes). This is of course not intended, and it would be nice to be able to detect those easily. I suppose that adding a 'Last build' column to the list view, that would display the time since the job entered its current state, would be nice. Something like 'Ended 8.6 hr' or 'Queued 1.3 hr' or 'Started 12 min'...
Then I'd know that if the code changed 3 hours ago, I shouldn't see any number larger than 3 hours...
Detect hung jobs
We have many jobs running, typically 20 to 30 simultaneously. And some builds last for several hours. It happens that tests hang, or are abnormally slow. These situations should be detected as soon as possible for investigation.
Unfortunately, the 'Build History' list is not very helpful, for two reasons. It has too few jobs for us: with 50 builds, only the last 5 hours are covered, which is less than the duration of many of our builds. But then if this limit was increased, we'd probably need a list of 200 or so jobs, which would not be easy to handle.
I would thus suggest to allow filtering on the 'building' status. When this flag would be set, the 'Build History' would only display the jobs that are currently being built.
A view 'by revision'
I often need to check if a given revision of the source has been built by a given job, or what is the latest revision that is good on a set of jobs. For example, I may want to merge this revision to some 'stable' branch for other teams to use.
I think that a grid view with the following attributes would be very useful for this: each line is a commit id or SVN revision, each column is a job, each cell is blue, red or gray (or even empty if this revision has not yet been part of a run of the job, or the run is not finished yet).
Do you think these would be useful additions?
Thursday, March 31, 2011
Partial working copies with Subversion
When working on large and/or multi-platform projects, it often happens that your Subversion working copy contains items that are useless: ports for other platforms, module that you don't need for your work, etc.
Recent Subversion versions allow to specify the maximum depth to use when updating a given target. You can e.g. specify that you only want the files in a given directory, but not its sub-directories. Or you want a given directory to be empty, or removed from the working copy entirely.
Let's consider you've checked out an entire working copy, with the following structure:
root
dir1
file1.1
file1.2
subdir1.1
subdir1.2
subdir1.3
subdir1.4
dir2
file2.1
subdir2.1
file2.1.1
dir3
file3.1
dir3.1
If you're not interested anymore in subdir2.1 and its content, you can issue the following commands:
> cd root/dir2
> svn update --set-depth exclude subdir2.1
Now dir2 only has file2.1, and an 'svn update' command that does not specify a '--set-depth' option will not bring back subdir2.1 or any of its content.
If you're only interested in the files inside dir1, but not the subdir1.*, you could do:
> cd root
> svn update --set-depth files dir1
Recent Subversion versions allow to specify the maximum depth to use when updating a given target. You can e.g. specify that you only want the files in a given directory, but not its sub-directories. Or you want a given directory to be empty, or removed from the working copy entirely.
Let's consider you've checked out an entire working copy, with the following structure:
root
dir1
file1.1
file1.2
subdir1.1
subdir1.2
subdir1.3
subdir1.4
dir2
file2.1
subdir2.1
file2.1.1
dir3
file3.1
dir3.1
If you're not interested anymore in subdir2.1 and its content, you can issue the following commands:
> cd root/dir2
> svn update --set-depth exclude subdir2.1
Now dir2 only has file2.1, and an 'svn update' command that does not specify a '--set-depth' option will not bring back subdir2.1 or any of its content.
If you're only interested in the files inside dir1, but not the subdir1.*, you could do:
> cd root
> svn update --set-depth files dir1
This tells SVN to only get the files in dir1, but not its sub-directories
Similarly, you can use '--set-depth empty' to get an empty directory (this can be useful to ask for one of its sub-directories), '--set-depth immediates' to get the immediate sub-directories, and use '--set-depth infinity' to restore the default and get everything in a given directory.
On the reverse, if you have to checkout a working copy but you know you only need some parts of it, you can use the '--depth' option (not '--set-depth', go figure...) of the checkout command. Then you create your paths in your working copy, step by step:
> svn checkout --depth empty [repository URL] root
> svn update --set-depth empty root/dir1
> svn update --set-depth infinity root/dir1/subdir1.2
> svn update --set-depth infinity root/dir3
Hope this helps!
Wednesday, January 5, 2011
Opening the Fisheye page for your current file from your IDE
Our revision control system is SVN, with a Fisheye server providing easy browsing of the repository. I use Fisheye very much and was tired of having to dig through the deep hierarchies of our project to display the information about a file. I therefore wrote a very little script that I can launch from Visual Studio to open the Fisheye page for the current file, with the correct branch/trunk information.
This script assumes that the directory structure on the disk mimicks the directory structure on the SVN repository, with a prefix (here: f:\work): f:\work\theProject\branches\aBranch\foo.txt corresponds to file foo.txt in the root directory of project theProject on branch aBranch. This will open page https://your-fisheye-server/browse/theProject/branches/aBranch/foo.txt. Of course, you will probably want to edit the location of the server...
You will have to launch this script from your IDE with as single argument the full path to the file you want to see in Fisheye. Here is how to do this from Visual Studio. Open the 'Tools' -> 'External tools...' menu, click the 'Add' button, and fill the information needed, using $(ItemPath) for the argument:
You now have a 'Current file on Fisheye' entry on your Tools menu, that you simply have to choose to launch Firefox...
I hope you'll find this useful...
This script assumes that the directory structure on the disk mimicks the directory structure on the SVN repository, with a prefix (here: f:\work): f:\work\theProject\branches\aBranch\foo.txt corresponds to file foo.txt in the root directory of project theProject on branch aBranch. This will open page https://your-fisheye-server/browse/theProject/branches/aBranch/foo.txt. Of course, you will probably want to edit the location of the server...
rem rem Launch a browser directed to the Fisheye page for rem a particular file. rem rem mailto:xavier.nodet@gmail.com rem @echo off rem Assumption: first argument is a absolute path to the file rem on the disk for which we would like to display the rem corresponding Fisheye page. set FILE=%1 rem Remove the "f:\work" prefix from the file path set FILE=%FILE:f:\work=% rem @echo %FILE% rem Replace all the backslashes with forward slashes set FILE=%FILE:\=/% rem @echo %FILE% start "c:\Program Files\Mozilla Firefox 4\firefox.exe" https://your-fisheye-server/browse%FILE%
You will have to launch this script from your IDE with as single argument the full path to the file you want to see in Fisheye. Here is how to do this from Visual Studio. Open the 'Tools' -> 'External tools...' menu, click the 'Add' button, and fill the information needed, using $(ItemPath) for the argument:
I hope you'll find this useful...
Subscribe to:
Posts (Atom)