When the build updates the packages; how does a developer ensure their environment is up to date?

Edit: There is now a NuGet PowerTools Package that will add a PackageConsole command to restore all packages.
I still like the TortoiseHook and still wish I could specify the packages location, but we’re getting closer to ‘there’.

Check In Packages

The obvious answer is to check in the packages. After all that’s how the SVN Externals solution worked and it ensures the repeatability tenant of continuous builds. Sometimes however it’s not an option for a number of reasons; company policy, ruthless team leads or dictating architects, take your pick.

Check In Packages to a “temp” repository.

The way our SVN externals solution worked was to keep a separate repository entirely for “packages”. This repository could be blown away when it got bloated and replaced by a new version. Because it was used for multiple solutions/Company projects, it also enabled remote workers to keep their own copy of the repository locally (using DNS or host file entries).

This method can still be used with NuGet and means that an SVN update updates everything.

Pre-Build Step

A pre-build step can be added for each project to run NuGet Install. I don’t really like this as it is an unnecessary build step. This task is an “Update” task not a build task and the speed of my build is sacrosanct to me.

Manually with Batch File

After updating the local copy, the Developer can run a batch file manually to update packages.

for /r %cd% %%X in (*.csproj) do (nuget install %%~pX\packages.config -OutputDirectory packages)

This batch file command runs nuget install for every packages.config in a solution.

Automate the Batch File with Tortoise Hooks

Obviously an automated solution is preferred to adding a secondary step to the developer’s workflow. If your VCS client supports hooks such as TortoiseSVN client hooks  you can check-in the above batch file and get developers to hook it in to their VCS Update.

Configure NuGet to use shared packages repository

The true Holy Grail for people who don’t want to check-in packages is the ability to tell NuGet where packages live. It could then be configured to use a shared network drive to find the packages. This would be the same repository used by the build so by default it is always up to date. Unfortunately this is not yet a feature of NuGet. It is being discussed as a new feature, so if its something you are keen on go vote for it at http://nuget.codeplex.com/workitem/215

The Build

The automated build for each component has 4 main build steps

1. Update and Install packages
   This is done by adding the Alpha NuGet Build Step and selecting the options relevant to your update strategy. I make liberal use of Build Parameters here.
2. Increment Version Number
   This is done via the TeamCity.targets.
   It takes the TeamCity build number, makes it the revision number in the AssemblyInfo. The pack command later uses this as the package version.
3. Build
4. Check In
   This is not implemented in the example and will depend on your VCS. However updating packages will change the packages.config and the project files of your solution. These will need to be checked in. If you also check in packages, these will need to be added as well.
5. Create, Lock Versions, Publish Package
   Creating and Publishing a package are trivial with NuGet. Package.Targets contains targets that Exec NuGet to do create a package. I use a Shared Drive for my repository so publishing is just a matter of specifying the output directory of pack to the repository UNC path. However there is also a publish command in NuGet if you want to push to the official feed.The interesting part of this step is my need to ensure all dependencies read from my project file will be locked to the version used to build the current version. This involves changing all dependency elements in the nuspec file inside the package. The target unzips the package, runs the LockDepedencyVersions Target found here and then re-zips the package.

Triggering

The last piece of the TeamCity puzzle is triggers; we want a build to be triggered if there is a new dependent package. At the moment it is a little clumsy as you must create a trigger per package you wish to monitor. I only monitor internal dependencies and then only direct internal dependencies.

So for example the UI project whilst it uses Common, Common is a dependency of Service A and Service B. So I only need to add triggers for those two packages. Because when Common changes, Service A and B will be triggered and when they publish their new versions the UI build will be triggered.

1. Updating of its dependencies for building
   

   Should we always update to the latest versions of our dependencies before building?
   Or only up to the latest minor version? Or Major version?
   Should we specify ranges for each individual package that we should update to?

   Nuget provides two options here, one is too specify (by hand ATM) the allowedVersions attribute in the packages.config (following the Version Specification). The update command then only updates to versions that are within the specified range.

   At the moment, the Nuget pack command doesn’t copy these across into the package. So this needs to be automated manually.
   (This should be fixed in the next version)

   The other option is to use the –safe option of Update. From the docs “Looks for updates with the highest version available within the same major and minor version as the installed package.”.

   In other words, if you have Elmah 1.0.0.0 installed, and 1.0.1.0, 1.0.2.0, and 1.1.0.0 are in the feed:

   Command	Description
   Update-Package Elmah	Updates Elmah to 1.1.0.0
   Update-Package Elmah -Safe	Updates Elmah to 1.0.2.0

    

2. Specifying allowable dependency versions during packaging?
   

   Should we restrict consumers to only using the exact version of our dependencies that we built with?
   Or should we allow any new version to be included?

   At the moment NuGet Pack adds dependencies from your project file with no version restrictions. So any consumers of your package can upgrade to newer versions of your dependencies. Any restriction you require will need to be done via custom build step to customise the dependencies or manually maintain a nuspec template that pack will use.

   I have created an MsBuild target called LockVersionDependencies . Which will lock dependency versions to the exact version used to build the package.

These are interrelated issues;  regardless whether your package is happy to upgrade to the latest of every dependency, those dependencies may restrict the allowable versions of other dependencies to an exact version or a restricted range.

My Preference

I prefer to allow all dependencies to update to the latest version during the build. If it’s successful I lock the dependencies to the exact version I built with when I create the package; so downstream builds will only use a combination of versions I know have built and passed the tests.

If a dependency updates and breaks the build, I would then fix the issue, or specify a version restriction for it so it doesn’t upgrade in future builds.

As per this diagram I have created a set of basic .Net projects that represent the problem and put it in a bitbucket repository.

The starting point is Tag, Version 1.

The Sample

The sample consists of 4 separate visual studio solutions, one for each component; the build directory that contains the MsBuild targets; the local Nuget Repository and some Content files.

Installation

1. Clone Tag – https://bitbucket.org/AdamMills/diamondsarentforever/src/Version1
2. Add the Local Repository to Nuget Settings or Update Repository Path in Build/All.Targets
3. Run PrepareEnvrionment.bat

The Local Repository

So the build doesn’t regularly download megabytes of packages (I don’t like storing them in VCS) I use a local repository. There are scripts on the interweb to mirror the full NuGet repository, but that seems too much for this little project. There is a nugget extension that can be used to copy individual packages, see here (note if using XP, you’ll need to manually add the LocalAppData Environment Variable)

With this I just maintain and checkin a batch file that downloads each package I need, so I can move or restore the local repository as needed.

I imagine that in the future Nuget will allow you to specify a cache, and then all team members can point to a common cache. Until then….

As above, a Diamond Dependency is when the dependency graph of a package forms a diamond shape.  It makes upgrading components towards the bottom of the graph more difficult as dependencies have to picked so all package and dependency versions match or are compatible.

For a more concrete example, taking the diagram above, say there is a new nifty feature in Common we want to use in the UI. ServiceA 2.0 is built with Common 2.0, however there is no ServiceB 2.0, so to upgrade to Common 2.0 in the UI we must either upgrade ServiceB, abandon all hope, or determine if Common 2.0 can be brought in side by side with Common 1.0. The latter solution can cause a lot of issues if types in Common are shared between communication with UI and ServiceB.

The general solution is then to “level” the dependencies i.e. upgrade everything to lowest common denominator. In the above solution; the lowest common denominator is Common 1.0, so we must use Service A and Service B 1.0, as well as using Common 1.0 in UI. When Service B 2.0 is released we can upgrade everything to 2.0

For a better explanation of the issue, see here. http://www.well-typed.com/blog/9

Previously, for me, the solution to packaging and levelling in .Net was to use another platforms packaging tools such as Maven or use SVN Externals to bring in dll’s and manually update the externals property, working out the levelling myself. With the introduction of Nuget the .Net world now has a package manager which can do the levelling for you, and with a little bit of work lets you automate and integrate the build process of numerous packages; regardless of diamond dependencies.

This is a series of posts; broken into the following:

• The Example Projects
• Picking an Update Packages Strategy
• The Build Process

Black-Scholes sample here.

I recently discovered that you can follow certain naming conventions to automagically get your targets imported into the build. Now, I know some people think automagic is even less transparent than explicitly defining something, however the naming is quite straightforward and obvious.

Add a targets file named; [after|before].<SolutionName>.sln.targets to the same directory as your solution file.

The after file is added after the project files and the before file is added before the project files.

Visual Studio has am inbuilt code-generator/templating engine!
http://t4talk.codeplex.com/

Can use it to generate compile-time and run-time code, or generate anything template based like config.

Seems like a good fit for high performance CQRS.

There is also a .Net port being done here.

Lambdas have some value outside performance that must be weighed, but if your app is sluggish and uses a lot of bindings; something to consider.