Mark Gilbert's Blog

Science and technology, served light and fluffy.

Come in ground control: Stratoballoon sketch updated for radio transmitter

It’s been a long time coming, but the main Balloon sketch has finally been updated to include code to control the radio transmitter.  And true to form, the transmitter had what I hope are the final few instrument pack monkey wrenches for us.

I detailed in Episode 22 the work needed to get the NTX2 board attached to the instrument pack.  Once I had that in place, I could incorporate the logic for transmitting over it from the RTTY test radio sketch we had been playing with for the last few months.

Monkey Wrench #1

Before I could drop those pieces in, however, I needed to relearn how to do string concatenation.  I wanted to transmit my call sign, a timestamp, altitude, latitude and longitude in a single comma-delimited string.  These values are a string, and four floating point values, respectively.  If this were straight C, I could use sprintf and format the floating point values to whatever precision I needed.  However, the Arduino C libraries don’t support sprintf for floats.

I could, however, use another built in function call "dtostrf()" which would convert my floats into a character array.  From there, I could use sprintf to concatenate the strings together into my "TransmitBuffer".

Monkey Wrench #2

Doing that, however preserved all of the whitespace in the character arrays.

KD8VZA,            21515.00,              314.28,              0.0000,              0.0000,*9395

It’s at points like these where I realize how spoiled I’ve gotten using a language like C# that supports things like ".Trim()".  Unfortunately, Arduino C doesn’t, so I had to roll my own.

What I ended up with was a routine that would copy one array into another, one byte at a time:

// This copies the SourceBuffer into DestBuffer starting at DestIndex,
// but trims out any leading spaces in SourceBuffer
int TransferBuffer(char* SourceBuffer, int SourceBufferSize, char* DestBuffer, int DestIndex, boolean ShouldAddComma) {
  int BufferIndex = 0;
 
  // Skip all characters from 0-32, but don’t count carriage returns, chr(13)
  while (BufferIndex < SourceBufferSize-1
          && SourceBuffer[BufferIndex] <= 32 && SourceBuffer[BufferIndex] != 13) { BufferIndex++; }
  while (BufferIndex < SourceBufferSize-1) {    
    DestBuffer[DestIndex] = SourceBuffer[BufferIndex];
    DestIndex++;
    BufferIndex++;
  } 
 
  if(ShouldAddComma) {
    TransmitBuffer[DestIndex] = ‘,';
    DestIndex++;
  }

  return DestIndex;
}

The TransmitBuffer method takes the source and destination arrays, the size of the source array, and the current position in the destination array where it needs to start writing.

The method starts by skipping past any whitespace or non-printable characters in the source array, except for carriage returns (character 13).

Once I know where in the source array I need to start copying, I copy the bytes into the destination array one by one, until I reach the end of the source array.  In that main loop, I’m keeping track of where I am in the destination array – this index will be the function’s return value.

The very last thing I do it check to see if I should append a comma to the end of the new string.  That is determined by the last parameter to the TransferBuffer array, ShouldAddComma.

Putting Monkey Wrenches 1 and 2 together

My original version of the TransmitDataToGround function uses these two solutions – dtostrf and TransferBuffer – to combine the call sign, timestamp, altitude, latitude, and longitude into a single string that can be transmitted over the NTX2:

void TransmitDataToGround(float TimeStamp, float Altitude, float Latitude, float Longitude) {
 
  // Build the string to transmit
  dtostrf(TimeStamp, 20, 3, TimeStampBuffer);
  dtostrf(Altitude, 20, 3, AltitudeBuffer);
  dtostrf(Latitude, 20, 5, LatitudeBuffer);
  dtostrf(Longitude, 20, 5, LongitudeBuffer);

  int TransmitIndex = 0;
  TransmitIndex = TransferBuffer(CALL_SIGN, sizeof(CALL_SIGN), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(TimeStampBuffer, sizeof(TimeStampBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(AltitudeBuffer, sizeof(AltitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LatitudeBuffer, sizeof(LatitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LongitudeBuffer, sizeof(LongitudeBuffer), TransmitBuffer, TransmitIndex, true);

  unsigned int CHECKSUM = gps_CRC16_checksum(TransmitBuffer);  // Calculates the checksum for this datastring
  sprintf(TransmitBufferChecksum, "*%04X\n", CHECKSUM);
  TransmitIndex = TransferBuffer(TransmitBufferChecksum, sizeof(TransmitBufferChecksum), TransmitBuffer, TransmitIndex, false);
 
  rtty_txstring (TransmitBuffer);
}

The logic for calculating the checksum, and the rtty_txstring() function which does the actual transmission, come from the UKHAS site.

Monkey Wrench #3

Once I had TransmitDataToGround() in place, I needed to determine how frequently to call it.  I ultimately decided that I would transmit every 10th reading (where a "reading" is a sampling of the temperature, pressure, and GPS sensors).  I updated the last section of the main loop logic (which checked that it had a good temperature and GPS reading before logging the reading) to include this call. 

// Now that we have a good GPS and temperature reading, grab
// the rest of the data, and log it.
if(HasGoodGPSReading && HasGoodTempReading)
{
  CurrentPressure = GetPressure(bmp085ReadUT(), bmp085ReadUP());
  CurrentAltitude = CalculateAltitude(CurrentPressure);
 
  PrintToSerialOutput(CurrentTimeStamp, CurrentTemp, CurrentPressure, CurrentAltitude, CurrentLatitude, CurrentLongitude);
  WriteDataToLogger(CurrentTimeStamp, CurrentTemp, CurrentPressure, CurrentAltitude, CurrentLatitude, CurrentLongitude);

  if(TransmitNumber > TRANSMIT_EVERY_Nth_READING) {
    TransmitDataToGround(CurrentTimeStamp, CurrentAltitude, CurrentLatitude, CurrentLongitude);
    TransmitNumber = 0;
  } else {
    TransmitNumber++;
  }
 
  HasGoodGPSReading = false;
  HasGoodTempReading = false;
}

 

I added a new loop index called "TransmitNumber" that I would increment with every reading, and reset when I invoked TransmitDataToGround().

All appeared to be fine at first – the data was coming through exactly as I had expected.  However, I noticed that the radio wasn’t ever turning off in between transmissions.  During the nine readings that weren’t being transmitted, it would generate a solid tone.  I needed to be able to programmatically enable and disable the transmitter.  That led me to rewire the NTX2 – for full details, see the Episode 22 Addendum.

Once I had digital pin 5 wired up properly, I reworked TransmitDataToGround() to turn the transmitter at the beginning, and turn it off again at the end.

void TransmitDataToGround(float TimeStamp, float Altitude, float Latitude, float Longitude) {
 
  // Build the string to transmit
  digitalWrite(RADIO_ENABLE_PIN, HIGH);

  dtostrf(TimeStamp, 20, 3, TimeStampBuffer);
  dtostrf(Altitude, 20, 3, AltitudeBuffer);
  dtostrf(Latitude, 20, 5, LatitudeBuffer);
  dtostrf(Longitude, 20, 5, LongitudeBuffer);

  int TransmitIndex = 0;
  TransmitIndex = TransferBuffer(CALL_SIGN, sizeof(CALL_SIGN), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(TimeStampBuffer, sizeof(TimeStampBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(AltitudeBuffer, sizeof(AltitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LatitudeBuffer, sizeof(LatitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LongitudeBuffer, sizeof(LongitudeBuffer), TransmitBuffer, TransmitIndex, true);

  unsigned int CHECKSUM = gps_CRC16_checksum(TransmitBuffer);  // Calculates the checksum for this datastring
  sprintf(TransmitBufferChecksum, "*%04X\n", CHECKSUM);
  TransmitIndex = TransferBuffer(TransmitBufferChecksum, sizeof(TransmitBufferChecksum), TransmitBuffer, TransmitIndex, false);
 
  rtty_txstring (TransmitBuffer);
  digitalWrite(RADIO_ENABLE_PIN, LOW);
}

Monkey Wrench #4

I found that when I enabled the transmitter, I was losing several characters off of the beginning.  This was because dl-fldigi needed 2-3 seconds between when the transmitter booted up until it actually started sending data to find the signal again, and start decoding it.  I added a 3-second delay to TransmitDataToGround, right after I enabled the transmitter, to provide this buffer.

void TransmitDataToGround(float TimeStamp, float Altitude, float Latitude, float Longitude) {
 
  // Build the string to transmit
  digitalWrite(RADIO_ENABLE_PIN, HIGH);
  delay(3000);
  dtostrf(TimeStamp, 20, 3, TimeStampBuffer);
  dtostrf(Altitude, 20, 3, AltitudeBuffer);
  dtostrf(Latitude, 20, 5, LatitudeBuffer);
  dtostrf(Longitude, 20, 5, LongitudeBuffer);

  int TransmitIndex = 0;
  TransmitIndex = TransferBuffer(CALL_SIGN, sizeof(CALL_SIGN), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(TimeStampBuffer, sizeof(TimeStampBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(AltitudeBuffer, sizeof(AltitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LatitudeBuffer, sizeof(LatitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LongitudeBuffer, sizeof(LongitudeBuffer), TransmitBuffer, TransmitIndex, true);

  unsigned int CHECKSUM = gps_CRC16_checksum(TransmitBuffer);  // Calculates the checksum for this datastring
  sprintf(TransmitBufferChecksum, "*%04X\n", CHECKSUM);
  TransmitIndex = TransferBuffer(TransmitBufferChecksum, sizeof(TransmitBufferChecksum), TransmitBuffer, TransmitIndex, false);
 
  rtty_txstring (TransmitBuffer);
  digitalWrite(RADIO_ENABLE_PIN, LOW);
}

Now when the radio turns on, the waterfall in dl-fldigi looks like this:

10

The blank horizontal line in the waterfall above shows this 3-second delay.  The red line that starts out horizontal, and then quickly curves to be vertical is the data coming over the airwaves.  That red line, and the accompanying one to the left, need to be lined up with the two thin red lines that make up the dl-fldigi cursor.  Until those two pairs of lines line up, nothing is decoded. 

I also found that the transmitter will drift slightly while it is transmitting, and that the dl-fldigi cursor will drift in between transmissions.  The 3-second delay gives me a chance to adjust the cursor if I need to when the next transmission starts.

Monkey Wrench #5

Even with the delay, I found I was still losing characters off the beginning of the transmission string.  I solved this by simply appending more than one copy of my call sign to the beginning.  As a result, the transmitted strings usually ends up looking like this now:

jf7&VZA-KD8VZA-KD8VZA,21515.00,314.28,0.0000,0.0000,*9395

Finally, I adjusted the squelch control in dl-fldigi to avoid having so much garbage between transmissions appear in the output window.  I still get some, but it’s not the constant barrage that it was when I started.

***

It feels good to be able to put this piece of the balloon to rest.  I may end refactoring the code a little, or adjusting it based on our instrument pack endurance/distance test, but otherwise I think its basically there.           

You can get the newly completed sketch from GitHub

July 8, 2014 Posted by | Science | Comments Off

Science Podcast, Episode 22 – Stratoballoon Radio Part 2

In this episode, Mark finally finishes the balloon’s antenna.

http://markofquality.wordpress.com/2014/07/01/science-podcast-episode-22-stratoballoon-radio-part-2/

July 1, 2014 Posted by | Podcast, Science | Comments Off

Stratoballoon Radio Details

Towards the end of our last podcast, I did a bit of hand-waving when it came to how we sent and received the test beacon data over the radio.  I quickly walked through the setup of the major pieces of software involved, but skipped over all of the details.  Today, I’ll be walking through those weeds.

On the transmission side, we have the second sample sketch available from the UKHAS site (link: http://TinyURL.com/UKHAS-NTX2 ) running on the Arduino sending the test beacon data through the NTX2 transmitter.

On the receiving side, there are actually three pieces of software at work:

  1. SDR#, pronounced "SDR-sharp", which is used to pull the raw radio signals off of the NooElec receiver
  2. dl-fldigi, which is used to translate the audio signals back into data
  3. VB-Audio Virtual Cable, which as the name implies is a virtual audio cable between SDR# and dl-fldigi

When I first got the SDR in the mail, I didn’t try transmitting the test beacon data right away.  My first test was actually to get the base software installed and try tuning in a local radio station.  For that test, I found some extremely helpful comments on the Amazon page that I purchased it from.  Those instructions involved going to NooElec’s site, downloading the Zadig driver installer and SDR# from there.  I did, and was in very short order up and running.  Here are the exact steps I took:

  • Went to NooElec’s site, and the product page for the R820T SDR Receiver: http://www.nooelec.com/store/sdr/sdr-receivers/tv28tv2-sdr-dvb-t-usb-stick-set.html#.UzbdOahdXNk
  • Went to Downloads, and downloaded the SDR# AutoInstaller
  • Extracted the downloader, and ran the Install.bat that came with it
  • Plugged in the USB stick. When Windows prompted me to install drivers, I cancelled it.  I can’t remember if it asked me to do this more than once, but if it did, I cancelled both attempts.
  • I ran Zadig.exe.  This is one of the EXEs that was installed to the /sdrsharp folder created by Install.bat.  It should have had WinUSB selected by default. I clicked "Install Drivers".  I closed Zadig when it finished.
  • I plugged in the antenna.
  • I ran SDR#.
  • To initially test it, I first selected "RTL-SDR / USB" from the drop down (my device), then clicked "Configure".

10
And configured it as follows:

20
I think the key things that I tweaked on this dialog were the Sample Rate and the RF Gain.

  • I clicked Close to return to the main screen.
  • I set the frequency to 107.7 MHz, a local rock station, checked "WFM" (Wide FM) and hit "Play". 

30

That worked – I could hear the radio station coming in fairly clearly, even in my basement, and I could then use the mouse wheel to change frequencies to other radio stations.  I could also change it manually.

Now that I had the receiver and SDR# working, it was time to try it out with the NTX2.  To decode the data, we would be using dl-fldigi, which was a free download.  While the source code is available off of Github, there are installers and binaries available off of the UKHAS site here: http://ukhas.org.uk/projects:dl-fldigi.  Once that was installed, I fired it up.

I initially spent 15 minutes or so trying to figure out a way to feed the output of SDR# directly the input of dl-fldigi, but didn’t have any luck.  I began to wonder if I needed something else to bridge the gap.  I found yet another page on the UKHAS site that described just such a bridge: VB-Audio

After I installed this, I rebooted my machine, and then followed the rest of the instructions found here:

  • Go into Control Panel / Hardware and Sound / Sound / Recording
  • You should see an input called Virtual Audio Cable.  Click properties and rename it to SDR.

40

  • Click the "Listen" tab
  • Check the "Listen to This Device" checkbox

50

  • Close the dialog
  • Stop SDR# and under the Audio section select "[MME] CABLE Input (VB-Audio Virtual C".  (I also played around with the main settings, and ultimately landed on a Filter Type of "Blackman-Harris 7" and a Filter bandwidth of "2400".  I honestly don’t know how much of a difference this makes yet.)

60

  • I clicked start, but had to do a couple of things before I could hear the test beacon through my speakers:
    • I had to change the receiver from WFM mode to USB, or "Upper Side Band"

70

    • I had to position the red line just to the left of the signal, otherwise it wouldn’t work.

80

    • Even though the NTX2 reports that it was transmitting at 434.650 MHZ, I could only really pick up the signal if I tuned the receiver to 434.635 MHZ.
  • Next, I started dl-fldigi again.  I clicked Op Mode / RTTY / Custom / Audio / PortAudio / Capture, and selected "CABLE Output (VB-Audio Virtual" from the drop down.  I clicked Close to return to the main screen.

90

Before I could actually see the test beacon correctly, I had to be sure that dl-fldigi was configured to decode it.  The instructions for this are described in http://TinyURL.com/UKHAS-NTX2, and the actual configuration is done under Op Mode / RTTY / Custom / Modems:

95

Finally, once I had everything configured, I had to adjust where the two red "data" lines appeared in the waterfall trace by tuning SDRSharp.  Once I could see that, I could match the two red "tuning" lines with the data lines, and actually start seeing data appear in the data box.

100

I found a few different ways that the tuning lines and the data lines wouldn’t match up:

First, if the two in the latter were too far apart, they wouldn’t match up with the data lines in the waterfall trace.

110
To fix that, I needed to adjust the carrier shift (otherwise, I wouldn’t see the data coming through).

120

 

Second, if the red lines were lined up with the signal in the waterfall, I would only see gibberish in the data window.

130

To fix that, I simply clicked the left data line in the waterfall to force the tuning line to match it.  The other line would stay in lockstep, always the same distance away (as determined by the carrier shift).

Finally, I also found that as I tuned SDR# up or down, the data lines in dl-fldigi would slide left or right.  If I let the signal slide away from the SDR# band completely, it would disappear from the waterfall trace in dl-fldigi.  This is something I need to keep in mind if I have problems getting the data to come in properly when we were out in the field trying to track the balloon while it was in flight.

There you have it.  The basic setup for our transmission and receiving station.  We still have a long way to go to getting the NTX2 fully incorporated into the instrument cluster, but Katherine and I feel we at least have a good foundation now.

May 8, 2014 Posted by | Science | Comments Off

Pixie Password Policy

One of our recent movie acquisitions was The Pirate Fairy, the fifth in Disney’s "Fairies" series, which shows Tinker Bell’s origins and develops the entire Pixie Hollow universe.  Don’t let the fact that these have never been in the theater* fool you – these movies have good stories, have great characters, are well-cast, and they’re funny!

"The Pirate Fairy" is about a dustkeeper named Zarina.  Pixie dust, you see, is what enables the fairies (and anything else) to fly, and it’s up to the dustkeepers to manage and distribute it.  Up until this movie, we know about two major kinds of dust – the golden pixie dust that you see getting sprinkled on everything, and a much rarer blue dust.  The blue dust is so rare and precious, in fact, that the dustkeepers actually keep it in a locked vault.  All of the dustkeepers apparently have the combination because one of the earliest scenes is of Zarina pulling "blue dust duty", which involves her and her boss, Fairy Gary, opening the vault to take some of the blue dust out for use.

Now, Zarina is not just any dustkeeper.  She’s an inquisitive dustkeeper.  A VERY inquisitive dustkeeper.  In her spare time, Zarina has been running a few, shall we say, "unsanctioned" experiments with pixie dust, trying to learn more about what it can do.  When one of her experiments gets away from her, Fairy Gary fires her.  She packs up her things, and leaves Pixie Hollow.

A year later, she returns during a festival where all of the fairies are gathered at the stadium, and puts all of them to sleep using poppy pollen – all except our heroes, of course.  She then proceeds to open the vault and steals all of the blue pixie dust.  I won’t ruin the rest of the movie for you, but it does have a happy ending.

***

What I’m going to focus on, though, is actually Fairy Gary and his lack of good password policies.  Let’s start with the sheer number of people that know the combination to the vault holding probably the most precious thing in all of Pixie Hollow.  How many dustkeepers do you have in your employ, Gary?  Do they ALL really need full admin rights?

And then there’s the fact that you haven’t change the combination to the vault in at least a year.  Even if Zarina hadn’t left, good password hygiene demands that a combination like this should be rotated every couple of months – at the minimum.

But then you fired an employee with full admin access.  That event alone should have triggered an immediate change to the combination.

And do I really need to even ask?  We’re only talking ONE combination to be changed here.  How much effort does it really take to change and distribute the new combination, really?

Sigh.  All this trouble could have been thwarted had Fairy Gary just used some basic system administration best practices.

 

* Well, almost never – accordingly to IMDB, Tinker Bell and the Lost Treasure apparently spent a week in the theaters in LA, just long enough to qualify for the 2010 Oscars.

May 6, 2014 Posted by | General | Comments Off

Science Podcast, Episode 21 – Stratoballoon Radio Part 1

Mark and Katherine try to cut through the gibberish and static, and get their radio transmitter working:

http://markofquality.wordpress.com/2014/05/03/science-podcast-episode-21-stratoballoon-radio-part-1/

May 3, 2014 Posted by | Podcast, Science | Comments Off

Resisting the woodpeckers – Builder Pattern

If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.
– (Gerald) Weinberg’s Second Law

A couple of months ago, I found a recommendation on my company’s intranet for "Growing Object-Oriented Software, Guided by Tests" by Steve Freeman and Nat Pryce.  I am an avid reader, but believe or not, I haven’t actually managed to read all the books yet, so I checked this one out from the library, and went through it. 

One of the real gems I found inside was a pattern I hadn’t seen before – the builder pattern (for those of you who have the book, this appears starting on page 258).  The goal here is to create data for a test in a way that keeps your test clean, but also gives you the flexibility to change that data when you need to.  It is also very expressive – you know exactly what part of the data is being changed just by reading the setup code.

Their example, which is what sold me on this pattern looks like this:

new AddressBuilder()
       .build();

This would get you a simple address, with (presumably) default, valid, values.  Next, the builder class would allow you to modify the defaults using methods like so:

new AddressBuilder()
       .withStreet("221b Baker Street")
       .withStreet2("London")
       .withPostCode("NW1 6XE")
       .build();

The methods withStreet(), withStreet2(), and withPostCode() would override the address 1, address 2, and postal code, respectively.  What’s more, this is far clearer than writing a method like UpdateTestData("221b Baker Street", “London”, "NW1 6XE") to do the same thing – is the second parameter modifying the street address or the city?  You no longer know at a glance, you now have to dig into the code to find out.

I’ve had to do things like this in the past numerous times, and my setup code for this has been wearisome-at-best to maintain.  Not only is this pattern clean and expressive, but it could start out simple and grow with my tests.  I could start out with the default case, and if I found that I later needed to test a case involving a custom postal code, I could add a method that allowed me to inject that value.  I wouldn’t have to touch any of my other tests, or do anything crazy with my setup logic – I would just have to add a method, and chain it for the one test that needed it.

I vowed that my next project would use this pattern, and in the last couple of weeks, I got the opportunity to put it to good use.  I was building a mechanism for NLog that would allow me to configure it from a database (rather than from web.config/app.config; I won’t go into detail on how this is done, but it turns out to be rather straightforward: https://github.com/nlog/NLog/wiki/Configuration-API ).  I would pass in a Dictionary of name-value pairs for the properties, and then I wanted to test that the proper NLog target and logger was configured (out of the gate I wanted to support a database target and a mail target).

After some back and forth – some of which was me getting comfortable with the pattern, and some of which was me letting the tests drive the functionality that I needed – I arrived at the following structure:

private class NLogSettingsBuilder
{
    private List<KeyValuePair<String, String>> _Settings;

    public NLogSettingsBuilder()
    {
        this._Settings = new List<KeyValuePair<String, String>>();
    }

    public NLogSettingsBuilder WithAllDatabaseSettings()
    {
        this._Settings = new List<KeyValuePair<String, String>>();
        this._Settings.Add(new KeyValuePair<String, String>("Target.Database.connectionStringName", "SomeName"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Database.commandText", "insert into EventLog ([Origin], [LogLevel], [Message], [StackTrace], SourceMachine) values (@origin, @logLevel, @message, @stackTrace, @sourceMachine);"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Database.name", "database"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Database.Parameter.origin", "Services"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Database.Parameter.logLevel", "${level:uppercase=true}"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Database.Parameter.message", "${message}"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Database.Parameter.stackTrace", "${stacktrace}"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Database.Parameter.sourceMachine", "${machinename}"));

        this._Settings.Add(new KeyValuePair<String, String>("Logger.Database.minlevel", "Error"));
        return this;
    }

    public NLogSettingsBuilder WithAllMailSettings()
    {
        this._Settings = new List<KeyValuePair<String, String>>();
        this._Settings.Add(new KeyValuePair<String, String>("Target.Mail.name", "email"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Mail.subject", "Blah Local Dev Error"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Mail.to", "mgilbert@blah.com"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Mail.from", "nlog@blah.com"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Mail.smtpServer", "smtp.blah.com"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Mail.encoding", "UTF-8"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Mail.body", "Timestamp: ${longdate}${newline}Level: ${level:uppercase=true}${newline}Logger: ${logger}${newline}Machine Name: ${machinename}${newline}${newline}Message: ${message}${newline}${newline}Stacktrace: ${stacktrace}"));
        this._Settings.Add(new KeyValuePair<String, String>("Target.Mail.html", "true"));

        this._Settings.Add(new KeyValuePair<String, String>("Logger.Mail.minlevel", "Error"));
        return this;
    }

    public NLogSettingsBuilder WithoutSetting(String Key)
    {
        this._Settings.RemoveAll(setting => setting.Key == Key);
        return this;
    }

    public NLogSettingsBuilder WithThisSettingAltered(String Key, String NewValue)
    {
        this.WithoutSetting(Key);
        this._Settings.Add(new KeyValuePair<String, String>(Key, NewValue));
        return this;
    }

    public Dictionary<String, String> Build()
    {
        Dictionary<String, String> NewSettings = new Dictionary<String, String>();
        if (this._Settings != null)
        {
            foreach (KeyValuePair<String, String> CurrentPair in this._Settings) { NewSettings.Add(CurrentPair.Key, CurrentPair.Value); }
        }
        return NewSettings;
    }
}

That allowed me to test things like:

this._NLogSettings = (new NLogSettingsBuilder())

                               .Build();

The default case, where there are no properties configured, and therefore no NLog targets will be configured.

this._NLogSettings = (new NLogSettingsBuilder())

                               .WithAllDatabaseSettings()

                               .Build();

All of the correct database settings will be present, so I should expect the database target and logger to be configured.

this._NLogSettings = (new NLogSettingsBuilder())

                               .WithAllMailSettings()

                               .Build();

All of the correct mail settings will be present, so I should expect the mail target and logger to be configured.

this._NLogSettings = (new NLogSettingsBuilder())

                               .WithAllDatabaseSettings()

                               .WithoutSetting("Target.Database.connectionStringName")

                               .Build();

All of the correct database settings will be present except for "Target.Database.connectionStringName".  Since this is a required property, I should not expect the database target and logger to be configured.

this._NLogSettings = (new NLogSettingsBuilder())

                               .WithAllDatabaseSettings()

                               .WithThisSettingAltered("Target.Database.connectionStringName", "Blah")

                               .Build();

All of the correct database settings will be present, and "Target.Database.connectionStringName" will have the value of "Blah".  I should not expect the database target and logger to be configured, and I should be able to test the connectionStringName property and confirm that its value matches "Blah".

As I said before, as soon as I read this, I was hooked.  I’ve struggled in the past to keep my test data manageable, and have had to bit a lot of bullets to make sweeping changes to keep it up to date.  This kind of pattern will help that immensely, and will go a great way, I think, to keeping the digital woodpeckers at bay.

April 23, 2014 Posted by | Visual Studio/.NET | Comments Off

Account, Development, and the Tension Between

Debbie: "You sold what?!"

Steve: "What do you mean you can’t build that?!"

Debbie: "You told the client we’d have it ready when?!"

Steve: "You can’t spend three weeks polishing the code – we need to go out tomorrow!"

If you are a developer or an account person in the software/digital marketing industry for any length of time, you will probably encounter or make some variation of the above statements.  What I’ve come to realize is that there is a fundamental tension between the account person – let’s call him Steve – and the development person – let’s call her Debbie. 

The typical Debbie can wax poetic about the ridiculous deadlines, insane requirements, and unbelievable client requests that account commits her to.  The typical Steve can do the same for the infuriatingly obstinate and short-sighted developers that he has to drag kicking and screaming through a project.  So this tension is bad, and we need to find ways to reduce and even eliminate it.  Right?

Wrong.

This tension serves an absolutely vital purpose in a company.  It mustn’t be banned – it just needs to be balanced.  To do that, we need communication.  Let’s say that our two personas, Steve and Debbie, actually go into business together and create a company called Spelltronix*.  We’re going to explore the tension between the two of them in three scenarios.

 

Scenario 1
Let’s say that Steve committed Spelltronix to building a new web site that does X, Y, and Z for A dollars in B weeks, without ever consulting Debbie.  Debbie is a more than a little offended when he brings back those details and says "Here you go".  Here are just a few problems Debbie might find with this scenario:

  1. Feature X is not feasible in the time allotted.
  2. Feature Y will cost more to develop than what you negotiated, or will require the client to buy licenses for third-party software for more than what you negotiated.
  3. I’m already working on Projects J and K, and therefore can’t start on this new one for another month, so the deadline is unreasonable.
  4. I have no expertise in this domain, so Feature Z is beyond me right now.  I’ll need to attend some classes, or do some extra reading to get up to speed, and that will require additional time.

My guess is Debbie won’t enjoy working with Steve for very long if he keeps committing them to projects like this.

 

Scenario 2
Now let’s say that Debbie agreed to build a system with requirements X, Y, and Z, for A dollars in B weeks.  Steve gets the agreement signed by the client, and Debbie goes off and starts work on the new system.  Steve comes back after B weeks to see how things have gone.  Debbie reports:

  1. Feature X is complete, Y is mostly done but buggy, and Z has not yet been started.
  2. Debbie spent a lot more time trying out new third-party libraries for feature Y, finding several that did the job, but most weren’t "neat and clean" in her opinion.
  3. She didn’t have any domain expertise with Feature Z, so she has been doing a lot of reading about it, trying to get up to speed.  That’s part of the reason why she hasn’t started it yet.
  4. She’s logged a lot of overtime, and spent a lot in licensing on those third party libraries, charging all of it back to the client.  As a result, costs on this project have already exceeded the A dollars that they bid, and that doesn’t include the time needed to finish the unstarted Feature Z.
  5. It will take another 2 weeks to finish up work on Feature Y, and then start and finish feature Z.

My guess is Steve won’t enjoy working with Debbie for very long if she keeps executing projects like this.

In both of these scenarios, the tension is very one-sided.  There was no balance.  What would help avoid these?  Project Managers, you say?  Maybe, but when you boil the role of a PM down, what does he or she really do on a project?  And no, the answer is not "Gantt charts".  A PM’s primary job function is make sure all of the stakeholders in a project have the information they need, when they need it, in order to keep the project on track, on budget, etc.  In a small shop like Spelltronix, I wouldn’t start by adding a PM.  I would start by getting Steve and Debbie to ACTUALLY TALK TO EACH OTHER! 

 

Scenario 3
Steve and Debbie visit a new prospective client together.  They spend a couple of hours gathering information about the client’s needs, asking questions about the new proposed system, and finding out where the client is headed in the future.  They return to their office, and spend the rest of the afternoon pouring over their notes individually.  They spend much of the next two days discussing the project together, where several questions consume the conversation:

  1. Do we think we understand all of the major requirements well enough to bid on this?
  2. How much do we think this will cost us to do?
  3. Do we have the necessary expertise?
  4. Can we build this in the timeframes that the client is requesting?
  5. Can we hire or contract pieces of this out?
  6. What about Projects J and K – can we wrap those up before we start this one, or even run all three concurrently for a period of time without getting into trouble?

They work up the proposal together, and are eventually awarded the work.  They bring on two contractors for the specific piece of the project that Debbie is not familiar with.  Debbie is able to manage the contractors while she wraps up her other two projects, and then joins them full-bore when those wind down.  Steve, Debbie, and the contractors hold daily meetings to coordinate efforts.  Debbie evaluates several third party libraries to use for one of the features, and while she doesn’t like how any of them look, the team all agrees that several will do the job.  To meet the deadline she selects the one she feels is the best of the bunch and allows them to keep things moving along.  The contractors are released when their piece is finished, and Debbie wraps up Phase 1 on time, without any major surprises.

In the first scenario, Steve is committing their company to things that Debbie can’t do for one reason or another, but if he had waited and talked through those proposed commitments with Debbie, they could come up with more sensible ones to propose. 

In the second scenario, Debbie got overzealous with her approach with the design and implementation of the solution, and needed to be reined in.  She also needed to say she needed help sooner on the feature that she had no expertise in.  Having regular touch points between Steve and Debbie (weekly-, or even daily stand-ups) would have kept her on task, and kept the project moving forward.

In both cases, more communication was needed between Steve and Debbie.  The lack of communication caused the tension between the two sides of the company to be one-sided, and therefore unbalanced.  That led to unbalanced results.

However, when the two sides communicated as in Scenario 3, the tension was balanced.  Steve and Debbie had to contend with all of the same issues in the third scenario – with all of the same tension.  They undoubtedly had to discuss and go back and forth on the issues.  Debbie couldn’t magically conjure up more time in a day to get her existing work done AND take on this new project.  Steve couldn’t magically convince the client to cut out the feature that Debbie wasn’t familiar with.  But because they were communicating, that tension was balanced, and they were able to work through those issues, and arrived at a much better result.

Is communication and balancing tension between someone like Steve and someone like Debbie going to ensure unicorns and rainbows at the end of the project?  Of course not.  But if Steve and Debbie DON’T communicate, the tension between them will be unbalanced from the start, which will make it more likely that dumb luck and 11th-hour heroics will be required to prevent their project from going off the rails.  And that’s no way to run a project, a team, or a company.

 

*I just made this up.  Any resemblance to a real company, living or dead, is purely coincidental.

April 11, 2014 Posted by | Software Process | Comments Off

The X Conference is coming

Registration for the 6th annual Kalamazoo X Conference opened today! 

Are you a knowledge worker?  Can you rip out IoC frameworks in your sleep?  Do you know every Excel shortcut there is?  Can you dazzle in T-SQL?  There are a plethora of technical conferences for you to hone your skills with.

The X Conference is different.  The X Conference is for all of the other skills you need to become a GREAT knowledge worker.

Join me on Saturday, April 26 at the Fetzer Center on the central campus of Western Michigan University in Kalamazoo, Michigan.  Mike Eaton and his exception crew have put what promises to be another mind-bending day together.  There is also chance to win one of three free tickets to this year’s conference by playing the Session Matchmaker Ticket Giveaway.  Check out the site for full details.

February 1, 2014 Posted by | Kalamazoo X Conference | Comments Off

Reducing the Tedium: Redux

Back in July, I wrote “Reducing the Tedium: Generalized Unit Tests via Reflection” which introduced a generalized unit test for strings and IDs in a pre-defined list of classes.  This was my first stab at trying to automate some of the more tedious tests I typically write for properties of this sort.  With my latest project, I was able to extend this even further with a new “PropertyValidator” class.

My first task was to eliminate the need for the explicit “TypesToCheck” array that defined which classes to test.  Since a lot of my use cases was checking business classes in the same assembly, I started by building a method that could take the name of the assembly to test as a parameter, open that assembly up via Reflection, and find all of the classes in it.

I started by repackaging my original code into two main functions – one for validating strings:

private static void ValidateStrings(Type CurrentType)
{
    String TestValue, ClassName;
    PropertyInfo[] ClassProperties;
    Object ClassInstance;
    
    ClassName = CurrentType.Name;
    ClassProperties = CurrentType.GetProperties().Where(p => p.PropertyType == typeof(String) && p.GetSetMethod() != null).ToArray();
    ClassInstance = Activator.CreateInstance(CurrentType);

    foreach (var PropertyUnderTest in ClassProperties)
    {
        if (ShouldSkipProperty(PropertyUnderTest)) { continue; }

        TestValue = (String)PropertyUnderTest.GetValue(ClassInstance, null);
        Assert.IsEmpty(TestValue, String.Format("{0}.{1} did not initialize properly", ClassName, PropertyUnderTest.Name));

        PropertyUnderTest.SetValue(ClassInstance, null, null);
        TestValue = (String)PropertyUnderTest.GetValue(ClassInstance, null);
        Assert.IsEmpty(TestValue, String.Format("{0}.{1} did not handle null properly", ClassName, PropertyUnderTest.Name));

        PropertyUnderTest.SetValue(ClassInstance, "", null);
        TestValue = (String)PropertyUnderTest.GetValue(ClassInstance, null);
        Assert.IsEmpty(TestValue, String.Format("{0}.{1} did not handle an empty string properly", ClassName, PropertyUnderTest.Name));

        PropertyUnderTest.SetValue(ClassInstance, "  ", null);
        TestValue = (String)PropertyUnderTest.GetValue(ClassInstance, null);
        Assert.IsEmpty(TestValue, String.Format("{0}.{1} did not handle a blank string properly", ClassName, PropertyUnderTest.Name));

        PropertyUnderTest.SetValue(ClassInstance, "abc123", null);
        TestValue = (String)PropertyUnderTest.GetValue(ClassInstance, null);
        Assert.AreEqual("abc123", TestValue, String.Format("{0}.{1} did not handle a valid string properly", ClassName, PropertyUnderTest.Name));
    }
}

 

And the other for validating IDs:

private static void ValidateIDs(Type CurrentType)
{
    long TestValue;
    String ClassName;
    PropertyInfo[] ClassProperties;
    Object ClassInstance;

    ClassName = CurrentType.Name;
    ClassProperties = CurrentType.GetProperties();
    ClassInstance = Activator.CreateInstance(CurrentType);

    foreach (var PropertyUnderTest in ClassProperties.Where(p => IsIDToValidate(p)))
    {
        if (PropertyUnderTest.GetCustomAttributes(typeof(ObsoleteAttribute), true).Count() > 0) { continue; }

        TestValue = (long)PropertyUnderTest.GetValue(ClassInstance, null);
        Assert.AreEqual(0, TestValue, String.Format("{0}.{1} did not initialize properly", ClassName, PropertyUnderTest.Name));

        PropertyUnderTest.SetValue(ClassInstance, 0, null);
        TestValue = (long)PropertyUnderTest.GetValue(ClassInstance, null);
        Assert.AreEqual(0, TestValue, String.Format("{0}.{1} did not handle being set to 0 properly", ClassName, PropertyUnderTest.Name));

        PropertyUnderTest.SetValue(ClassInstance, -1, null);
        TestValue = (long)PropertyUnderTest.GetValue(ClassInstance, null);
        Assert.AreEqual(0, TestValue, String.Format("{0}.{1} did not handle being set to a negative properly", ClassName, PropertyUnderTest.Name));
    }
}

 

These do the same basic checks on each string or ID property as described in the July post.  They make use of three helper functions – ShouldSkipType, ShouldSkipProperty, and IsIDToValidate.  I’ll describe these in turn:

private static bool ShouldSkipType(Type CurrentType)
{
    return CurrentType.GetCustomAttributes(typeof(PropertyValidator.Attributes.DoNotPerformBasicValidation), true).Length > 0 
        || CurrentType.IsGenericType 
        || CurrentType.IsAbstract
        || CurrentType.FullName.StartsWith("PostSharp");
}

The ShouldSkipType function returns True if the class:

  • is a generic type
  • is an abstract type
  • starts with “PostSharp” (this is a logging library that I started using this summer, and its classes are added to mine post-build; trying to run my test code against these classes cause problems, so I just skip them entirely)
  • has the custom “DoNotPerformBasicValidation” attribute attached.

If any of these conditions is met, the class is skipped.  I found a need to create exceptions to the rule of “check every class in this assembly”, so I created the “DoNotPerformBasicValidation” attribute that I could decorate a specific class in an assembly to be skipped.  This attribute has no logic of its own – it is merely used as a flag on the class:

using System;
using System.Collections.Generic;
using System.Linq;

namespace PropertyValidator.Attributes
{
    [AttributeUsage(AttributeTargets.All)]
    public class DoNotPerformBasicValidation : Attribute 
    { 
    }
}

This attribute can be applied to not only classes, but also individual properties, and the ShouldSkipProperty function looks for that:

private static bool ShouldSkipProperty(PropertyInfo CurrentProperty)
{
    return CurrentProperty.GetCustomAttributes(typeof(PropertyValidator.Attributes.DoNotPerformBasicValidation), true).Length > 0;
}

This function returns True if the custom attribute is found on the current property being evaluated.

Finally, to determine if a given property is a string, “ValidateStrings” merely looks at the property’s type.  To determine if it is an ID to be checked is a little trickier.  I can’t rely solely on its base type, so instead I require that property to be treated as an ID is marked as such, with another custom attribute:

using System;
using System.Collections.Generic;
using System.Linq;

namespace PropertyValidator.Attributes
{
    [AttributeUsage(AttributeTargets.Property)]
    public class ValidateAsID : Attribute 
    {
    }
}

When the “ValidateAsID” attribute is applied to a property, the “IsIDToValidate” function returns true:

private static bool IsIDToValidate(PropertyInfo CurrentProperty)
{
    return CurrentProperty.GetCustomAttributes(typeof(PropertyValidator.Attributes.ValidateAsID), true).Length > 0;
}

 

The “ValidateStrings” and “ValidateIDs” functions need a list of classes to open and examine.  That list is provided by the PropertyValidator’s “Validate” methods:

public static void Validate(String AssemblyPartialName)
{
    Validate(AssemblyPartialName, null);
}
public static void Validate(String AssemblyPartialName, String ClassName)
{
    Assembly AssemblyToValidate;
    Type[] AssemblyTypes;

    AssemblyToValidate = Assembly.Load(AssemblyPartialName);
    if (AssemblyToValidate == null) { throw new Exception(String.Format("Could not load {0}", AssemblyPartialName)); }

    AssemblyTypes = AssemblyToValidate.GetTypes();

    foreach (Type CurrentType in AssemblyTypes)
    {
        try
        {
            if (ShouldSkipType(CurrentType)) { continue; }
            if (ClassName != null && CurrentType.Name != ClassName) { continue; }

            System.Diagnostics.Trace.WriteLine(String.Format("Now testing '{0}'", CurrentType.Name));

            ValidateStrings(CurrentType);
            ValidateIDs(CurrentType);
        }
        catch (Exception ex)
        {
            throw new Exception(String.Format("Error validating the type '{0}'", CurrentType.Name), ex);
        }
    }
}

The first takes only the assembly name, and will attempt to check every class in that assembly.   The second will allow you to check a single specific class in an assembly.  For each class being checked, then, the method will invoke both “ValidateStrings” and “ValidateIDs”.

To use it, then, I can do something as simple as

PropertyValidator.Validate(“MyAssembly”);

or

PropertyValidator.Validate(“MyAssembly”, “MyClass”);

***

PropertyValidator gives me a much better testing framework for properties than before, and is very easy to use.  I still have a dependency on NUnit, however, and it doesn’t do anything with other data types such as datetimes.  Perhaps the next iteration.

December 23, 2013 Posted by | Visual Studio/.NET | Comments Off

Two-word searches

Years ago I came across a game you could play with Google where you tried to find a two-word search that would turn up exactly one result.  The fun part of this game was that once you found one of these magical searches, you’d post it somewhere on the ‘net, and then after a little while Google would come along, crawl your post, and then *poof*, there would be two results for that pair of search terms – the original and your port – thus making your find very short-lived.  I tried a few crazy searches of my own, seeing if I could stumble upon one of these, but I wasn’t ever able to.

Fast forward several years.  I was troubleshooting an NHibernate issue I was working on, and I did a search using two terms, and it came up with exactly one hit:

Google Results

I ran this search on December 10, and at the time the link returned a 404.  Google has updated since then to remove the search result, so re-running the search for these two terms turns up 0 hits now.

Short-lived, true, but hey, I finally found one!

December 20, 2013 Posted by | General | Comments Off

Follow

Get every new post delivered to your Inbox.