My buffer runneth over September 7, 2009
Posted by markegilbert in Tools and Toys, Visual Studio/.NET.add a comment
In the new release of NAntRunner, one of the new features is a checkbox to add the -verbose switch to the NAnt call. In the course of testing it, I found that NAnt would hang when I ran a non-trivial build and deploy script. When NAnt.exe hung, it locked up NAntRunner, too. If I ran that same script from the command line using NAnt (bypassing NAntRunner), it completed fine.
Through some tinkering I found that once NAnt.exe hung, I could kill it off using Task Manager and then NAntRunner would give me control back. I started doing some searching for “process.start hung” and “process.start doesn’t exit”, and eventually came to a couple of articles that talked about the output buffers filling up:
Both of these discuss redirecting the buffers for a spawned process. As it turns out, the buffers have a 2K limit and if they fill up, the process hangs. That sounded awfully familiar.
All versions of NAntRunner have redirected the standard output and error buffers for the spawned process so I could display the contents of those buffers in the NAntRunner interface. Before 0.4, I would wait until the process finished and then get the entire contents of the buffer:
Class NAntProcess
…
Public ReadOnly Property StandardError() As String
Get
Return Me._SystemProcess.StandardOutput.ReadToEnd
End Get
End Property
Public ReadOnly Property StandardError() As String
Get
Return Me._SystemProcess.StandardOutput.ReadToEnd
End Get
End Property
…
End Class
When I started using the verbose switch, the NAnt process hung because the process generated much more output than before (imagine that, “verbose” equals “more”), and the output buffers filled up before the process finished. So, I had to change how I was grabbing the output messages off.
The VelocityReviews.com link above mentions BeginOutputReadLine, so I did some more digging and came up with this MDSN article which shows how to pull these messages off asynchronously:
http://msdn.microsoft.com/en-us/library/system.diagnostics.process.beginoutputreadline.aspx
This involved handling the OutputDataReceived and ErrorDataReceived handlers for the process, and then appending the messages as they were generated using a pair of StringBuilder objects. The modifications looked like the following (only the relevant portions of the NAntProcess class are shown):
Class NAntProcess
Private _StandardOutputBuilder As StringBuilder
Private _ErrorOutputBuilder As StringBuilder
Public Sub New(ByVal NAntExecutablePath As String, _
ByVal NewScriptToRun As String, _
ByVal NewTargetFramework As String, _
ByVal NewBuildTarget As String, _
ByVal ShouldEnableVerboseMessages As String, _
ByVal NewOtherArgs As String)
…
‘ Configure the asynchronous output collection
Me._StandardOutputBuilder = New StringBuilder
Me._StandardOutputBuilder.AppendLine(Me._NAntExecutable & ” “ & Me.GetArgumentsForNAnt)
Me._StandardOutputBuilder.AppendLine()
Me._StandardOutputBuilder.AppendLine()
Me._ErrorOutputBuilder = New StringBuilder
AddHandler Me._SystemProcess.OutputDataReceived, AddressOf Me.StandardOutputHandler
AddHandler Me._SystemProcess.ErrorDataReceived, AddressOf Me.ErrorOutputHandler
End Sub
Private Sub StandardOutputHandler(ByVal sendingProcess As Object, ByVal outLine As DataReceivedEventArgs)
If Not String.IsNullOrEmpty(outLine.Data) Then
Me._StandardOutputBuilder.AppendLine(outLine.Data)
End If
End Sub
Private Sub ErrorOutputHandler(ByVal sendingProcess As Object, ByVal outLine As DataReceivedEventArgs)
If Not String.IsNullOrEmpty(outLine.Data) Then
Me._ErrorOutputBuilder.AppendLine(outLine.Data)
End If
End Sub
Public ReadOnly Property StandardOutput() As String
Get
Return Me._StandardOutputBuilder.ToString
End Get
End Property
Public ReadOnly Property StandardError() As String
Get
Return Me._ErrorOutputBuilder.ToString
End Get
End Property
End Class
After I got this implemented, I realized that I wouldn’t need the verbose switch to cause this problem. In theory, the “regular” output from a very involved NAnt script could cause the buffers to fill up. It’s just a fluke that I hadn’t hit this up to this point.
NAntRunner 0.4 Released September 4, 2009
Posted by markegilbert in Tools and Toys.add a comment
Meet the new kid
I just released NAntRunner 0.4, now available at http://CodePlex.com/NAntRunner. Upgrading to the new version is easy. Simply download the 0.4 ZIP, and extract it to the folder with your previous installation of NAntRunner (your existing settings will be preserved). There were three main things I wanted to tackle in this release. I ended up doing four.
First, I replaced the treeview controls with a user control that rolls everything up. This user control actually started out life in another one of my tools – IISTweak – and I had always intended to incorporate it into NAntRunner. When I did that I realized I had some more work to do to get it closer to being generic. I don’t think the control is completely baked yet, but it’s definitely closer.
For those of you who download the source code and want to tinker, I found an interesting quirk with Visual Studio 2008 and this update. The new user control is called NavigationTree, and it is part of the NAntRunner assembly (the executable). If you do anything with the control in the designer mode on the Main form, Studio updates the instantiation of this control in Main.Designer.vb file to read as follows:
Me.MainNavigationTree = New NAntRunner.NavigationTree
That, however, causes a compile-time error. If you remove the “NAntRunner” assembly reference so it reads as follows:
Me.MainNavigationTree = New NavigationTree
Then the compiler is happy again. Sigh. It’s the little things that cause the grey hairs.
The second update was to add a checkbox that will add the -verbose switch to the NAnt call. As you’d expect, the additional messages generated will appear in the NAntRunner progress window. In previous version of NAntRunner, you could include this yourself using the “Other NAnt Args” box, but having a checkbox feeds my lazy side.
Third, I added the ability to reorder scripts within a group and moving scripts from one group to another – all via drag and drop. I found myself having to scan down through the groups looking for the right script more and more recently, and really wanted to be able to move the more commonly used ones to the top of the list. In the past you could always hack the NAntRunnerSettings.xml file to manually reorder these, but this is much more intuitive. Reordering entire groups and their contents isn’t available at this time.
The one item I didn’t plan on but ended up doing was reworking how the standard output and error messages were retrieved from the spawned NAnt process. Previously it was done at the end, once the NAnt task had completed. Version 0.4 does this asynchronously via event handlers – as the messages are generated. The reasons why and the details of the implementation will be a subject for a future post.
The future
I started thinking about what would define the 1.0 release. My goal with NAntRunner has always been to provide easier access to the common features of NAnt. I think it’s almost there. I may extend the “verbose” checkbox into a “message level” control (perhaps a slider) so NAntRunner has native support for the -verbose, -debug, and -quiet switches. I’ve also thought about adding logging support (using the -logger switch). Finally, I may extend the interface to allow you to save the NAntRunner configuration settings used for a given build script so that the next time you open that script NAntRunner will configure itself accordingly (saving you a few clicks and keystrokes).
Minimally, I’ve dropped several TODOs in the code for things that I need to get back to. Since many of these involve error handling, these are a must for version 1.0. If you can’t rely on your tools then you need better tools.
Enjoy the new release, and let me know what you think.
Organizational System Fail August 24, 2009
Posted by markegilbert in General, Tools and Toys.2 comments
The core of my organizational system for many years has been Microsoft OneNote (most of that experience is with OneNote 2003, although I spent a year with OneNote 2007). In the last year I coupled that Microsoft Live Mesh, and the combination was awesome. I could organize my day and week from home, then walk into the office and find those notes synched up with my work machine.
It was easy to use. It was reliable. I didn’t have to think about it – it just worked.
And then on Friday, it didn’t.
I booted up my machine to find two projects’-worth of notes completely gone. Those two projects just happened to be my MAIN two projects currently – weeks of thoughts, ideas, questions and solutions: gone.
I actually found myself talking to my computer: “Oh no, you did NOT just do that.” Sigh. Oh yes, it did. What a way to start a Friday.
I managed to recover a few of the points from memory, but as Professor Jones would say “I wrote them down…so that I wouldn’t HAVE to remember.” More importantly, I resolved that this weekend I would solve the core issue. Namely – the combination of OneNote and Live Mesh, or more specifically, how OneNote stores its data.
The core of my organizational system was four OneNote “notebooks”. Each notebook contained one or more “pages” where each page contained the notes for a single project. The notebooks were named “Today”, “This Week”, “Work” and “Personal”, and my typical day would begin by moving pages into “Today” so I would have the focus for the day. As I completed them, I’d delete them. If they were put on hold for some reason, I’d move them to “This Week” or one of the others.
Live Mesh, for its part, is configured to watch one specific folder on my machine – the folder where I keep my .one files. If it sees one on my hard drive that’s more recent than the corresponding version in the cloud, it copies it up. If it finds one in the cloud that’s more recent than the corresponding one on my hard drive, it copies it down.
Simple right? Of course not. If it were simple I wouldn’t be writing a blog post about it.
My machine at home is on 24/7. That means that I can move a page from notebook A to B, and within a minute or so that move will be replicated to the cloud. My machine at work, on the other hand, it usually off when I’m not there. So, when I get into work in the morning, I boot up my machine and Live Mesh with it. Live Mesh then takes a minute or two to look at the cloud copies of my .one files, and if it finds any updates it pulls them down.
OneNote manages a single .one file for each notebook. As I moved a page from “Today” to “This Week”, for example, both of the corresponding .one files would be updated to reflect the move. That means that if I move a page from notebook A to notebook B, and then go into work, boot up, and change B at work before Live Mesh has had a chance to bring down the update to B from home, that update will be lost. The page won’t exist in A anymore (because I moved it out of there), and my work-update to B will overwrite the home-update to B (because the work-update is more recent).
I actually had this happen to me a couple of months ago. The damage then was a relatively minor and short-lived project (“schedule dental appointment”, or something like that), so it was easily recovered from memory. At the time I thought I learned my lessons:
1) When I come into the office in the morning, let Live Mesh boot up and synch up before trying to use OneNote locally.
2) When I am about to leave the office in the evening, let Live Mesh post my most recent updates to the cloud before shutting down.
As near as I can tell on Friday, I didn’t do anything with OneNote for the first 15 minutes my machine was running – I didn’t even have OneNote opened.
So, as I sat there is nearly stunned silence last Friday morning trying to will my notes back into existence, I resolved that I would do one of the following:
1) Find a way to store my notebook pages in separate files so simple moves wouldn’t cause entire projects to blink out of existence, or
2) Find another tool to keep my notes organized, or
3) Write a new tool that would meet my needs.
I did some digging into using OneNote and Live Mesh, and found many people raving about the combination, but nobody talking about any synching problems. The closest I got was this post on Microsoft Connect.
The other tools (task managers) that I found were way too UI-heavy: too many buttons, options, colors, etc. I just wanted something very subdued, easy, and flexible. I don’t need or want to subdivide my projects into 500 discrete steps. All I need is a text editor and the ability to indent – after all, that’s basically what I had been using OneNote for all these years.
So, that left me with rolling my own. What I ended up with was a simple desktop application that I call “Tasks”:
Tasks basically allows me to organize and manage a pile of text files – one per project – on the file system. Tasks stores all of the text files in one folder on the file system, and that folder is synched between my two machines via Live Mesh. Additionally, the task groups and there contents (Today, Tomorrow, etc.) are defined in an XML file, also in that same, synched folder. The result is that I see the exact same view at home or at work. Within Tasks I can create new tasks, move them from group to group, reorder them within a group, change the file name, open an editor to modify their contents, and delete them.
I haven’t decided if I’m going to release this as another tool or not (at the very least, it’s not ready for general release now). This was something I crammed in over the weekend, so while it does what I need it to do, there are a lot of little things that were left out due to time constraints. However, today was the first real day in action for Tasks, and it performed well.
For me, the tool to manage tasks has become such a critical application that it needs to be ridiculously dependable. Don’t get me wrong – this isn’t medical monitoring equipment or a spacecraft guidance system, it’s just a task manager. But, it’s that tool that allows me to be as organized and efficient as I am. If that tool isn’t easy to use, if it isn’t reliable, if I have think about it, then questions of “is my data safe” begins to creep in and sap that efficiency. Get those critical tools – task managers, source control, data backup, whatever they are – and work to get them as automatic and as reliable as breathing. The time and anxiety you save by doing that more than makes up for the investment.
IISTweak 0.1.1 Released October 29, 2008
Posted by markegilbert in Tools and Toys.add a comment
I just posted a minor update to IIS Tweak. This corrects an issue with the tree view where deleting the last item in the tree wouldn’t update the buttons or the context menu commands correctly. You can download the compiled binary or the source at http://www.codeplex.com/IISTweak.
SSMS Scripter Updated – Major Performance Improvement October 7, 2008
Posted by markegilbert in SQL Server, Tools and Toys.add a comment
I just released a new version of the SSMS Scripter – version 2.6.0 is now available at http://CodePlex.com/SSMSScripter.
The major update for this version was a huge (two-orders-of-magnitude-huge) improvement in the time needed for the Scripter to load up once invoked from SQL Server Management Studio. With version 2.5.3, even databases with a handful of stored procedures, views, or user-defined functions would take 100-200 seconds for the Scripter to spin up. It ended up being three lines of code to drop that load time to two seconds or less, even for large databases.
After doing some digging, I found this article by Bill Graziano dated November 29, 2005 (just about a year before I wrote the very first version of the Scripter): http://www.sqlteam.com/article/scripting-database-objects-using-smo-updated. This article was doing the same basic thing that I was – enumerating the non-system objects in a database via SMO. Bill’s sample code was referencing the same property I am – IsSystemObject – and found it to be very slow. Boy, this sounds familiar.
The reason for the lousy performance was explained by a member of the SQL Server Product Team, Euan Garden, who was responding to Bill’s post (a link to Euan’s comments is included in Bill’s article). By default, the IsSystemObject property is not returned when you get a collection of objects back (like stored procedures). As Euan explains, this was done to optimize how much SMO needs to retrieve with each request. As it turns out, you can force specific properties to be returned, though:
Me._CurrentServer.SetDefaultInitFields(GetType(StoredProcedure), “IsSystemObject”)
Me._CurrentServer.SetDefaultInitFields(GetType(View), “IsSystemObject”)
Me._CurrentServer.SetDefaultInitFields(GetType(UserDefinedFunction), “IsSystemObject”)
I added these lines to the newly created (as the result of a refactoring) “CurrentServer” property in SSMSScriptor.vb. This configures the SMO.Database object to bring IsSystemObject back as one of the properties of the StoredProcedure, View, and UserDefinedFunction objects. By not doing this, the code was having to make a round trip to the database for EVERY object in the collection before it could say whether or not it was a system object. This meant every stored procedure in the database, plus all of the ones in Master – thousands in all. And it ONLY took 200 seconds. Ahem.
In addition to the performance boost, I also updated the class headers throughout to reflect the license that the Scripter is being released under now, refactored the main interface class a bit, modified the interface so that it no longer closes once you finish scripting to a file or to the clipboard, and adjusted how the progress bar/status messages are displayed.
Enjoy!
New Tool Published: IIS Tweak September 28, 2008
Posted by markegilbert in Tools and Toys.add a comment
Standard operating procedure for me and my colleagues when working on a web site locally is to take over the default web site – http://localhost. However, we all work on multiple projects. Switching from one to another involves opening up the IIS management console, drilling down to “Default Web Site”, opening up the properties, clicking on the Home Directory tab, browsing out to the new folder, and then clicking OK a few times to save the new settings. I must have done this routine half a dozen times last week alone, not to mention how many times my teammates and I have done it in the past months.
Luckily, the tediousness of this process finally annoyed me to the point where I wrote a new utility to speed it up – IIS Tweak.
The first version, 0.1.0, uses the same treeview navigation controls that another of my utilities, NAntRunner, does. Each node on the list has a path to a web site’s root directory tied to it. When I double click an entry in the list, the utility will modify the IIS Metabase for web site with ID “1″, the default web site, and change the “Path” property to that path. This magic is the result of these four lines:
Dim MyLocalhost As DirectoryEntry
MyLocalhost = New DirectoryEntry(“IIS://localhost/w3svc/” & Me._WebSiteID & “/ROOT”)
MyLocalhost.Properties(“Path”).Item(0) = value
MyLocalhost.CommitChanges()
“Me._WebSiteID” in this case happens to be a private variable initialized to “1″ in the form’s constructor. If you know the ID for another virtual directory on the server, you could substitute it here and modify that site’s Home Directory, or any of a number of other properties. I’ve only done a little with the IIS Metabase, and the System.DirectoryServices namespace, but it looks like most of IIS (if not all) can be programmatically controlled; this is merely one simple example.
Full source code can be found on CodePlex: http://www.codeplex.com/IISTweak. Enjoy!
SSMS Scripter Republished to CodePlex.com September 6, 2008
Posted by markegilbert in SQL Server, Tools and Toys.add a comment
I’ve moved the source code and latest release (2.5.2) to CodePlex: http://CodePlex.com/SSMSScripter. This particular version was the last one released, and was done so here on my blog in February 2007.
When I first released it in late 2006 and early 2007, I wrote a series describing how the plug-in worked: http://Mark-Gilbert.com/category/sql-server/. In the “Part 2″ post in that series, I mentioned that I couldn’t find any decent documentation on how to write an SSMS plug-in. At the time I had hoped that my series would help form that documentation. Since then I’ve found another post at JCooney.net that goes into some additional depth from my series: http://jcooney.net/archive/2007/11/26/55358.aspx. As the link implies, Joseph Cooney wrote this in late 2007, almost a year after I wrote mine and he had a similar lament with a section of the post titled “So Why Does This Qualify as a Black Art?”. I agree with Joseph – writing one of these is definitely a black art, and I had to play a lot of code-roulette to get my plug-in to work.
For the next release, I have a performance-related update in mind, but I’m still sorting out how I might go about it. The actual process of generating the scripts is relatively quick (measured in seconds), regardless of the number of objects being scripted. The Scripter startup time can be substantial, especially for databases with large numbers of views/stored procedures/functions, and especially if you invoke the utility at the Database level in the SSMS hierarchy.
I also have not tried installing it with SQL Server 2008, and I’m eager to see how well (or not so well) it works there.
Let me know what else you’d like to see in the next release!
NAntRunner 0.3 Released August 30, 2008
Posted by markegilbert in Agile, Tools and Toys.add a comment
I just released NAntRunner 0.3, now available at http://CodePlex.com/NAntRunner. The biggest single change was dropping the list box of scripts, and introducing a tree view:
The top level of the tree view is a list of script “groups”. These can be called anything you like. Within each script group you can place one or more scripts. When you first add a script, the name that appears in the tree view will be the file name of the script itself, for example “Primary.build”. You can rename that to anything you like.
There are controls in the tree view context menu for adding groups, adding scripts, renaming groups/scripts, deleting groups/scripts, and running scripts. The five buttons that appear above the tree view replicate that functionality.
I’ve also added a “Last Saved” message next to the Close button in the lower right corner. Like its predecessors, NAntRunner 0.3 automatically saves all changes (adding or removing scripts, renaming items in the tree, resizing the utility, etc.). The message is simply there to let you know when it happens.
Upgrading to the new version is easy. Simply download the 0.3 ZIP, and extract it to the folder with your previous installation of NAntRunner. The 0.3 release will automatically upgrade the settings for the previous release. Any scripts that you had defined will be placed into a single group called “Default”. You can move them to new groups, and rename them at that point.
I had originally planned for a feature where you could drag and drop scripts to rearrange them, or move them to new groups, but I ended up postponing it until a later release. The tree view, and specifically being able to associate a nice display name with each script, solved the main annoyance with the 0.2 release – namely that if you had scripts at any significant depth on your file system, they became unreadable. Dropping and adding scripts is fairly easy with this release (simply copy the path from the “Path to Script” text box, hit the Add Script button, and paste it into the dialog), so I don’t see moving scripts/groups as a huge priority right now. If that’s a feature that you would find it hard to live without, please let me know in the comments here or on the CodePlex site.
Enjoy!
Tools and Toys Update August 23, 2008
Posted by markegilbert in SQL Server, Tools and Toys, WPF/Silverlight.add a comment
I’ve moved the source code for the WPF Media Player to CodePlex: http://codeplex.com/WPFMediaPlayer. Currently the Player sits at version 0.1, but I’ve been working on a version 0.2 for a while now. Ok, I was working on it earlier this year, and it’s been sitting dormant for a while now. Ahem. That should be changing in another week or so. I’m wrapping up a personal project of mine, and then I focus on a handful of programming projects that I’ve been putting off for a while – this being one of them.
The WPF Media Player is actually the second project I’ve moved to CodePlex – the first being NAntRunner, now hosted at http://codeplex.com/nantrunner. I have an update in mind for NAntRunner as well, which would bring it up to version 0.3. Hopefully that will be posted in the next month or so.
Now that I’ve moved these to CodePlex, I’ve been thinking about the other items still hosted on my Tools and Toys page:
- The SSMS Scripter seems like a natural to post there, so that one will probably be next to move. I have an update in mind for that one as well, but it’s not clear to me how to go about making it. I’ll have to give it some more thought first.
- In the four months since I wrote and started using String Cleaning, I’ve found it to be eminently useful, but the core of the application is the one line of source described in this post. I can’t really see it changing all that much, so I don’t think that’s worth setting up a new project for. I have considered porting it to a PowerShell script (which would hopefully decrease the cold-start time), but that’s pretty low on the priority list at the moment.
- One-Three Back Solitaire hasn’t changed in years, and it was really my learning application for Test-Driven Development. At this point I don’t think that will move over either. If I ever decided to add on to it (write a web-based leader board using WCF, or something like that) I’d probably move it over first, just to simplify things.
There you have it. I’ll keep you posted on progress on these projects, and I’d love to hear your feedback on what you find useful, and what you’d like to see in future releases.
NAntRunner 0.2 Released May 26, 2008
Posted by markegilbert in Agile, Tools and Toys.add a comment
I published an update to the days-old NAntRunner project on CodePlex. I had barely begun to use version 0.1 myself and had just showed someone else how to use it, when we discovered how much it was really lacking. Version 0.2 should address a lot of those shortcomings:
Natively, version 0.2 supports selecting a framework to target (pulled from the NAnt.exe.config file) as well as the build target (pulled from the script you select). In addition, you can pass any other NAnt arguments you like using the textbox at the bottom. The complete command executed is shown at the top of the results box, which has been moved to the right side of the screen.
Along with the updated utility, I’ve released the NUnit test suite that I’ve been compiling for the utility. I think it is a good start, but it is not complete. In fact if you scan through the NAntRunner source code you’ll see many TODO notes describing tests or additional validation checks needed.
Let me know what you think of the update, and what else you’d like to see in future versions.
