Mark Gilbert's Blog

Science and technology, served light and fluffy.

It’s January, time to bundle up! TeamCity, PowerShell, and SharpZipLib

Late last year we deployed a new TeamCity build server at Biggs|Gilmore, and part of that process involves building a ZIP file that contains the files and folders to be deployed in that release.  We decided to use PowerShell to script out this piece of the process, and it has worked out tremendously well for us.

When it came time to actually create the ZIP file, we found that the PowerShell Community Extensions had a cmdlet called "Write-Zip", as well as the companion cmdlet "Expand-Archive".  We got that installed, and found its usage to be very easy – just pipe in an arbitrary list of files to be ZIPed up.  Things seemed to be going well.

However, shortly after we started using the new TeamCity server for production releases, we noticed that about half the time we did a build, a small number of files (usually 1-3) would just not get deployed.  Those files were being included in the build server’s ZIPs, but they simply weren’t being extracted.  When I tried to manually extract the contents to my desktop using 7Zip, it would report that the problem files were actually corrupt (usually with a cyclical redundency check error).  The source files in Subversion were fine, but once they were run through the build server, they became corrupt.  Initially I thought it may have been caused by the overall size of the ZIP files being built, or the folder depth that the files sat in, but there were other ZIPs that were larger or had a deeper folder structure, and were just fine.

I found precious little in the way of error reports about corrupt files coming out of Write-Zip.  Last week I decided to drop Write-Zip, and find a different way to build ZIPs.

I started by going back to our older build server – CruiseControl.NET.  We were using NAnt there to script out the package-and-deployment step, specifically the <zip> task.  Under the covers, I found that <zip> uses SharpZipLib, a compression library written in C#.  I looked into how I could use that library within PowerShell to create the archives.

I came across a blog post by John Marquez titled "PowerShell SharpZipLib Script".  In it, John shows how to wrap SharpZipLib in a custom function that can take as input a list of files to be compressed.  This seemed perfect.  When I tried it locally, though, I was plagued with errors like these:

Exception calling "CommitUpdate" with "0" argument(s): "Access to the path ‘C:\Users\mgilbert\Desktop\MyProject\bin\en-CA’ is denied."

The errors were coming from the .commitUpdate() method on the ICSharpCode.SharpZipLib.Zip.ZipFile object.  John said his example was built using SharpZipLib 0.85.5.  I had downloaded a slightly updated version of the library (v0.86.0), so I tried grabbing the exact version he had used.  That one didn’t build an archive at all – no errors, but no ZIP files either.

I opened SharpZipLib up in Telerik’s JustDecompile, and found that there were a few other ways to create ZIP files, but none that looked like they could update an existing archive with new files – something that I would need to do if I wanted to pipe in files one by one.  My other option (which is basically how we were using NAnt’s <zip> task) was to bundle up an entire folder and its files and subfolders.  Since most of the deployments I do are incremental builds, this would mean I would would shuffle off all of the files I needed to include into a separate, "corral", folder, bundle that up, and then clear it out (readying it for the next build).  Not a horrible solution, especially if it worked.

So, I reworked Marquez’s "zip" function to take an arbitrary list of files, copy them to a separate folder, bundle that folder up into a ZIP, and then clear the corral out again.  The actual compression of the corral folder’s contents would be done using the FastZip class in the SharpZipLib library.  Here is what I ended up with:

    [System.Reflection.Assembly]::LoadWithPartialName("ICSharpCode.SharpZipLib")

    function BiggsZip($zipFile, $CorralFolder, $LeadingFolderPathToRemove)
    {
        begin
        {
            if (Test-Path $CorralFolder) {
                Remove-Item $CorralFolder -Recurse -Force
            }
        }
        process
        {       
            $RelativeFile = $_.FullName.Remove(0, $LeadingFolderPathToRemove.length)
            if(-not $_.PSIsContainer) {
                New-Item -ItemType File -Path (Join-Path $CorralFolder $RelativeFile) -Force
            }
            Copy-Item $_.FullName (Join-Path $CorralFolder $RelativeFile) -Force
        }
        end
        {
            $zip = New-Object ICSharpCode.SharpZipLib.Zip.FastZip
            $zip.CreateZip($zipFile, $CorralFolder, $true,"")
            Remove-Item $CorralFolder -Recurse -Force
        }
    }

This is designed to have a list of files piped in using something like Get-ChildItem.  For example:

Get-ChildItem "C:\MyProject\trunk\wwwroot" -recurse | BiggsZip "MyArchive.zip" "c:\Users\mgilbert\Desktop\Corral" "C:\MyProject\trunk\wwwroot"

For this function (which I eventually installed as a PowerShell module), the SharpZipLib library was registered in the GAC on the TeamCity server.  The function has three components, denoted by the keywords "begin", "process", and "end".  I’ll describe these in turn.

 

begin
This is executed once at the beginning, before it starts processing the file list.  In this case, if it finds that $CorralFolder currently exists, it deletes it.

process
This is executed once for each file in the set that is piped into the function.  The archive should preserve the files’ paths relative to the web root.  The web root is passed into the function as the $LeadingFolderPathToRemove variable.  So, the "process" step copies the current file from its present location to the same relative path below CorralFolder.

The "New-Item" command is designed to get around a rather annoying issue with Copy-Item.  I found that Copy-Item by itself will happily copy an entire folder with structure below it to a new destination, and creates that folder and structure if it doesn’t exist.  It is also fine copying individual files to a folder that already exists.  However it will not copy a file into a folder that doesn’t already exist, even if I use the -Force switch.

I found this post on StackOverflow that confirmed this limitation of Copy-Item, and two possible ways to get around it.  I opted for option 2, which was to "touch" the destination file using New-Item (in the Unix sense of the word “touch”).  New-Item does not have the same limitation that Copy-Item does, so it is perfectly content to create a new, 0-byte file, and will create the folder structure above it if it doesn’t exist.  After that, I can just copy the real file over top of that one.

The check that surrounds New-Item "if(-not $_.PSIsContainer)" translates to "if the current item is not a folder".  I found that trying to "touch" a folder results in an error when it went to try to do the copy:

Copy-Item : Container cannot be copied onto existing leaf item.

I realized later that what was probably happening was when New-Item came across a directory, it would create it as an extension file – that is, a “leaf item”.  Then, when Copy-Item tried to copy the directory over it, it fails.  Since I could already use Copy-Item to create new folders, I limit the use of New-Item to just creating files.

end
This is executed once at the end, after the function has processed the entire file list.  At this point, all of the files to be included in the release are now in the CorralFolder, so the only thing left to do is create the actual ZIP file.  It does this, and then removes the CorralFolder.

 

That builds the ZIP file beautifully.  I then used John’s "unzip" function as is (other than renaming it) to extract it:

    function BiggsUnzip($zipFile, $unzipToFolder)
    {
      $zip = new-object icSharpCode.sharpZipLib.zip.fastZip
      $zip.extractZip($zipFile, $unzipToFolder, $null)
    }

I wrapped these two functions up into a BiggsZip.psm1 module file, making both functions publically accessible (note the "Export-ModuleMember" call at the very end):

    [System.Reflection.Assembly]::LoadWithPartialName("ICSharpCode.SharpZipLib")

    function BiggsZip($zipFile, $CorralFolder, $LeadingFolderPathToRemove)
    {
    …
    }

    function BiggsUnzip($zipFile, $unzipToFolder)
    {
      …
    }

    Export-ModuleMember -Function *

I then dropped it into a new BiggsZip folder under %windir%\System32\WindowsPowerShell\v1.0\Modules on the TeamCity server.  When I started a new PowerShell session, all I needed to use this library was to do this first:

Import-Module BiggsZip

And then use the BiggsZip and BiggsUnzip functions like I was using Write-Zip and Expand-Archive previously.  For the process of creating the PowerShell module file, and getting it installed on the server, I consulted the following:

The new solution has only been in place for a few days, but every one of my test archives have come through with flying colors – not a single corrupted file.  This took me three days of off-and-on troubleshooting and tinkering, but I think the result (trustworthy builds, and furthering experience with PowerShell) was absolutely worth it.

About these ads

January 16, 2013 - Posted by | Powershell

Sorry, the comment form is closed at this time.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: