Mark Gilbert's Blog

Science and technology, served light and fluffy.

Science Podcast, Episode 28 – Robot!

In this episode, Mark and Lucy build a robot!

https://markofquality.wordpress.com/2015/02/14/science-podcast-episode-28-robot/

February 14, 2015 Posted by | Podcast | Comments Off

Science Podcast, Epsiode 27 – Stratoballoon Rigging

Mark and Katherine describe how the capsule, radar reflector, parachute, and balloon were all held together.

https://markofquality.wordpress.com/2015/01/19/science-podcast-episode-27-stratoballoon-rigging/

January 19, 2015 Posted by | Podcast, Science | Comments Off

Science Podcast, Episode 26 – Stratoballoon Capsule Construction, Part 2 of 2

Mark and Katherine finish the details of the stratoballoon capsule construction:

https://markofquality.wordpress.com/2015/01/14/science-podcast-episode-26-stratoballoon-capsule-construction-part-2-of-2/

January 14, 2015 Posted by | Podcast, Science | Comments Off

Science Podcast, Episode 25 – Stratoballoon Capsule Construction, Part 1 of 2

Mark and Katherine walk through Part 1 of the stratoballoon capsule construction:

https://markofquality.wordpress.com/2015/01/13/science-podcast-episode-25-stratoballoon-capsule-construction-part-1-of-2/

January 13, 2015 Posted by | Podcast, Science | Comments Off

Science Podcast, Episode 24 – Stratoballoon Capsule Recovery!

Mark and Katherine celebrate the recovery of the capsule with some amazing photographs, and Rainbow Dash’s debriefing:

http://markofquality.wordpress.com/2014/11/04/science-podcast-episode-24-stratoballoon-capsule-recovery/

November 4, 2014 Posted by | Podcast, Science | Comments Off

Will you just wait a minute?! NUnit and Async/Await

I was being a good-doobie.  Honest.

I had written a prototype – just something to prove that a particular approach could work.  It did, and so now it was time for me to actually bring that code up to production quality.  Part of that meant modifying how I was invoking the third-party web services – they  needed to be done asynchronously.

So I went through and wrote my unit tests to also run asynchronously:

        [Test]
        public async void Divide_4DividedBy2_Equals2()
        {
            AsyncUnitTest.Math MathLibrary = new AsyncUnitTest.Math();

            float Quotient = await MathLibrary.Divide(4, 2);

            Assert.AreEqual(2, (int)Quotient);
        }

I ran it through NUnit on my machine, and everything was peachy-keen.  I ended up writing nearly 50 unit tests* like that, converting over all of my calls.  I committed the code, and let TeamCity take over.

And watched every one of those new tests break.

When I looked at the TeamCity log, the errors seemed to hint that the test runner was simply not waiting for the thing under test to complete before trying to run the asserts.  I started searching for things like "nunit async", and pretty quickly across this two-part series by Stephen Cleary:

In this series, Cleary says that the underlying problem of running async tests is that they don’t have a proper context:

We’ve encountered a situation very similar to async in Console programs: there is no async context provided for unit tests, so they’re just using the thread pool context. This means that when we await our method under test, then our async test method returns to its caller (the unit test framework), and the remainder of the async test method – including the Assert – is scheduled to run on the thread pool. When the unit test framework sees the test method return (without an exception), then it marks the method as “Passed”. Eventually, the Assert will fail on the thread pool.

His solution is to simply give the test an async context, and he provides a very handy wrapper to do just that.  I first had to install his Nito.AsyncEx NuGet package, and then wrap my test in AsyncContext.Run:

        [Test]
        public void Divide_4DividedBy2_Equals2_Asynchrofied()
        {
            AsyncContext.Run(async () =>
            {
                AsyncUnitTest.Math MathLibrary = new AsyncUnitTest.Math();

                float Quotient = await MathLibrary.Divide(4, 2);

                Assert.AreEqual(2, (int)Quotient);
            });
        }

Notice that I’ve removed the "async" keyword from the test itself; AsyncContext.Run does all the work here.  After updating and committing my first test using AsyncContext.Run – a test test, if you will – it ran successfully on TeamCity.  I updated the other 48, and finally got a green build.

***

My build was stable again, but Cleary’s explanation didn’t answer the question of why this worked on my machine in the first place – without using his very awesome library – so, I kept digging.

I first looked up exactly what TeamCity was using to run the tests – it was NUnit, the same as what was on my machine, with a minor different in the version.  My local copy was 2.6.2, while the version on the build server was 2.6.1.  Could there be a difference in how 2.6.1 was handling async?

Why yes.  Yes there was.  In the NUnit 2.6.2 release notes I found this:

When running under .NET 4.5, async test methods are now supported. For test cases returning a value, the method must return Task<T>, where T is the type of the returned value. For single tests and test cases not returning a value, the method may return either void or Task.

- Source: http://nunit.org/index.php?p=releaseNotes&r=2.6.2

Are you serious?  I just happen to have the first version of NUnit that would properly handle async on my machine, but the build server was one notch older, and therefore couldn’t?  *facepalm*

To further prove that this was the real source of my issue, I installed NUnit 2.6.1 and 2.6.2 side by side on my machine.  I took my two tests from above, both of which should have tried to execute the MathLibrary.Divide function which included a 2-second delay:

    public class Math
    {
        public async Task<float> Divide(int Numerator, int Denominator)
        {
            await Task.Delay(2000);
            return Numerator / Denominator;
        }
    }

When I ran these two tests through NUnit 2.6.1, Divide_4DividedBy2_Equals2 completes in a couple hundredths of a second, while Divide_4DividedBy2_Equals2_Asynchrofied takes just over 2 seconds to complete, for a total of just over 2 seconds:

2-6-1

When I ran these through NUnit 2.6.2, EACH test takes just over 2 seconds to complete, for a total of just over 4 seconds:

2-6-2

So, I have two choices – switch my builds on TeamCity to use at least NUnit 2.6.2 to run the tests, or use Cleary’s Nito.AsyncEx library, which will allow me to leave the build server as is.  In any event, at least I have a reasonable explanation for what was happening. 

The funny thing is that it’s usually MSBuild that messes with me.  Apparently NUnit gave him the week off.

 


* Yes, I realize that by calling the service directly, this no longer counts as a "unit" test, but rather an integration test.  That distinction isn’t relevant to the issue described in this post, though, so I’m going to gloss over the mock objects in the real code.

October 23, 2014 Posted by | Agile, Visual Studio/.NET | Comments Off

Balloon Mapper

Our receiving station for the Stratoballoon project consisted of three major pieces of software:

I’ve gone into great depth about how the first two are configured on my technical blog (“Stratoballoon Radio Details”).  Today, I want to talk about the third one, BalloonMapper.  Here is the basic interface:

10

We would copy a dataline out of dl-fldigi and past it into the "Data Line" box (#1).  We could then hit Enter, or manually click the "Find Position" button in the upper right corner.  This would extract the altitude, and the lat/long from the data string.  It would convert the former into feet and display that in the "Altitude" box (#2).  It would also create the Google Maps URL that would show the lat/long as a pin on the map.

What did the data strings look like?  A picture-perfect data string that came off the radio would look like this:

    KD8VZA-KD8VZA-KD8VZA,120057.00,234.11,4214.7460,8533.3750,*4FF4

This is a comma-delimited string, and is made up of the following:

  • My callsign, repeated 3 times
  • A timestamp, in hhmmss.xx format (where the decimal places are ignored)
  • The altitude, in meters
  • The latitude (format explained below)
  • The longitude (format explained below)
  • A termination string, made up of an asterisk, followed by four alphanumeric characters

In most cases the callsigns came out a bit garbled, so it would look more like this:

    a8czZA-KD8VZA-KD8VZA,120057.00,234.11,4214.7460,8533.3750,*4FF4

The first part of the string got chewed up because it took us a second to tune dl-fldigi to lock onto the signal.  That’s the main reason I start the string with three copies – I wanted to give Katherine or I (whoever was working the radio) a second or two to get ahold of the signal.

Extracting the altitude was very straightforward.  Simply grab the 3rd piece of data in the string, multiple it by 3.28 to convert it from meters to feet, and display it in the box.

        public static String GetAltitude(String DataLine)
        {
            String[] DataLineComponents;
            String RawAltitude;

            DataLine = (DataLine ?? "").Trim();
            if (String.IsNullOrEmpty(DataLine)) { return ""; }

            DataLineComponents = DataLine.Split(',');

            RawAltitude = DataLineComponents[2];

            return String.Format("{0} ft", (double.Parse(RawAltitude) * 3.28).ToString("0"));
        }

The lat/long was a bit tricker.  First, I had to get them into a format that Google Maps would understand.  You can browse directly to a specific lat/long point on Google Maps like so:

    https://www.google.com/maps/place/42°17’44.76"N+85°43’22.50"W

The lat/long values, however, always come off the radio in the 4-dot-4 pattern.  Here is how they broke down:

    4214.7460 = 42° 14.7460′

    8533.3750 = 85° 33.3750′

So, I would need to split the degrees from the rest of the string, then convert the fractional arc-minutes into arc-seconds, before I could drop it into Google Maps:

        public static String GetUrl(String DataLine)
        {
            String[] DataLineComponents;
            String RawLat, RawLong, FormattedLat, FormattedLong;

            DataLine = (DataLine ?? "").Trim();
            if (String.IsNullOrEmpty(DataLine)) { return ""; }

            DataLineComponents = DataLine.Split(',');

            RawLat = DataLineComponents[3];
            RawLong = DataLineComponents[4];

            FormattedLat = String.Format("{0}°{1}'{2}\"", RawLat.Substring(0, 2),
                                                          RawLat.Substring(2, 2),
                                                          (double.Parse(RawLat.Substring(4)) * 60).ToString("00.00"));
            FormattedLong = String.Format("{0}°{1}'{2}\"", RawLong.Substring(0, 2),
                                                           RawLong.Substring(2, 2),
                                                           (double.Parse(RawLong.Substring(4)) * 60).ToString("00.00"));

            return String.Format("https://www.google.com/maps/place/{0}N+{1}W", FormattedLat, FormattedLong);
        }

So, now I had my URL.  I needed a way to embed a browser into BalloonMapper, which was a Windows desktop app.  I looked at a few options, but eventually settled on Gecko Effects: https://bitbucket.org/geckofx/geckofx-29.0/downloads.

I created a Gecko.GeckoWebBrowser object on my form called "GoogleMaps".  To browse to a URL, I simply called the .Navigate() method on the GoogleMaps object, and passed it the URL that I generated above:

                this.GoogleMaps.Navigate(GetUrl(this.DataLineBox.Text));

 

Using it was easy.  Getting the control installed and working was a little more difficult, and I had a few false starts.  In the end, here is what worked:

  • I downloaded Gecko Effects 29.0 (see link above).
  • Gecko requires another application called "xulrunner", which I obtained from here: http://ftp.mozilla.org/pub/mozilla.org/xulrunner/releases/29.0.1/runtimes/
  • I found that I needed to match the version of xulrunner to the version of Gecko, otherwise I got a "Specified cast is not valid" error.
  • I also found that the program has to be installed to a folder called "xulrunner" – all lowercase – or it wouldn’t work.

This made finding the capsule’s current location extremely easy:

20

For full source code and binaries, please visit http://TinyURL.com/MarkGilbertSource, and look for the BalloonMapper.zip archive.

October 10, 2014 Posted by | Science, Visual Studio/.NET | 1 Comment

Science Podcast, Episode 23 – Stratoballoon Eggbeater Ground Antenna

Mark and Katherine build an antenna.  Or mangle a kitchen utensil.  Or possibly both.

http://markofquality.wordpress.com/2014/10/06/science-podcast-episode-23-stratoballoon-eggbeater-ground-antenna/

October 6, 2014 Posted by | Science | Comments Off

Stratoballoon – Thank You

Mark and Katherine say a heartfelt "Thank You" to everyone who contributed to the Stratoballoon project.

http://markofquality.wordpress.com/2014/09/23/stratoballoon-thank-you/

September 23, 2014 Posted by | Podcast, Science | Comments Off

Come in ground control: Stratoballoon sketch updated for radio transmitter

It’s been a long time coming, but the main Balloon sketch has finally been updated to include code to control the radio transmitter.  And true to form, the transmitter had what I hope are the final few instrument pack monkey wrenches for us.

I detailed in Episode 22 the work needed to get the NTX2 board attached to the instrument pack.  Once I had that in place, I could incorporate the logic for transmitting over it from the RTTY test radio sketch we had been playing with for the last few months.

Monkey Wrench #1

Before I could drop those pieces in, however, I needed to relearn how to do string concatenation.  I wanted to transmit my call sign, a timestamp, altitude, latitude and longitude in a single comma-delimited string.  These values are a string, and four floating point values, respectively.  If this were straight C, I could use sprintf and format the floating point values to whatever precision I needed.  However, the Arduino C libraries don’t support sprintf for floats.

I could, however, use another built in function call "dtostrf()" which would convert my floats into a character array.  From there, I could use sprintf to concatenate the strings together into my "TransmitBuffer".

Monkey Wrench #2

Doing that, however preserved all of the whitespace in the character arrays.

KD8VZA,            21515.00,              314.28,              0.0000,              0.0000,*9395

It’s at points like these where I realize how spoiled I’ve gotten using a language like C# that supports things like ".Trim()".  Unfortunately, Arduino C doesn’t, so I had to roll my own.

What I ended up with was a routine that would copy one array into another, one byte at a time:

// This copies the SourceBuffer into DestBuffer starting at DestIndex,
// but trims out any leading spaces in SourceBuffer
int TransferBuffer(char* SourceBuffer, int SourceBufferSize, char* DestBuffer, int DestIndex, boolean ShouldAddComma) {
  int BufferIndex = 0;
 
  // Skip all characters from 0-32, but don’t count carriage returns, chr(13)
  while (BufferIndex < SourceBufferSize-1
          && SourceBuffer[BufferIndex] <= 32 && SourceBuffer[BufferIndex] != 13) { BufferIndex++; }
  while (BufferIndex < SourceBufferSize-1) {    
    DestBuffer[DestIndex] = SourceBuffer[BufferIndex];
    DestIndex++;
    BufferIndex++;
  } 
 
  if(ShouldAddComma) {
    TransmitBuffer[DestIndex] = ‘,';
    DestIndex++;
  }

  return DestIndex;
}

The TransmitBuffer method takes the source and destination arrays, the size of the source array, and the current position in the destination array where it needs to start writing.

The method starts by skipping past any whitespace or non-printable characters in the source array, except for carriage returns (character 13).

Once I know where in the source array I need to start copying, I copy the bytes into the destination array one by one, until I reach the end of the source array.  In that main loop, I’m keeping track of where I am in the destination array – this index will be the function’s return value.

The very last thing I do it check to see if I should append a comma to the end of the new string.  That is determined by the last parameter to the TransferBuffer array, ShouldAddComma.

Putting Monkey Wrenches 1 and 2 together

My original version of the TransmitDataToGround function uses these two solutions – dtostrf and TransferBuffer – to combine the call sign, timestamp, altitude, latitude, and longitude into a single string that can be transmitted over the NTX2:

void TransmitDataToGround(float TimeStamp, float Altitude, float Latitude, float Longitude) {
 
  // Build the string to transmit
  dtostrf(TimeStamp, 20, 3, TimeStampBuffer);
  dtostrf(Altitude, 20, 3, AltitudeBuffer);
  dtostrf(Latitude, 20, 5, LatitudeBuffer);
  dtostrf(Longitude, 20, 5, LongitudeBuffer);

  int TransmitIndex = 0;
  TransmitIndex = TransferBuffer(CALL_SIGN, sizeof(CALL_SIGN), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(TimeStampBuffer, sizeof(TimeStampBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(AltitudeBuffer, sizeof(AltitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LatitudeBuffer, sizeof(LatitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LongitudeBuffer, sizeof(LongitudeBuffer), TransmitBuffer, TransmitIndex, true);

  unsigned int CHECKSUM = gps_CRC16_checksum(TransmitBuffer);  // Calculates the checksum for this datastring
  sprintf(TransmitBufferChecksum, "*%04X\n", CHECKSUM);
  TransmitIndex = TransferBuffer(TransmitBufferChecksum, sizeof(TransmitBufferChecksum), TransmitBuffer, TransmitIndex, false);
 
  rtty_txstring (TransmitBuffer);
}

The logic for calculating the checksum, and the rtty_txstring() function which does the actual transmission, come from the UKHAS site.

Monkey Wrench #3

Once I had TransmitDataToGround() in place, I needed to determine how frequently to call it.  I ultimately decided that I would transmit every 10th reading (where a "reading" is a sampling of the temperature, pressure, and GPS sensors).  I updated the last section of the main loop logic (which checked that it had a good temperature and GPS reading before logging the reading) to include this call. 

// Now that we have a good GPS and temperature reading, grab
// the rest of the data, and log it.
if(HasGoodGPSReading && HasGoodTempReading)
{
  CurrentPressure = GetPressure(bmp085ReadUT(), bmp085ReadUP());
  CurrentAltitude = CalculateAltitude(CurrentPressure);
 
  PrintToSerialOutput(CurrentTimeStamp, CurrentTemp, CurrentPressure, CurrentAltitude, CurrentLatitude, CurrentLongitude);
  WriteDataToLogger(CurrentTimeStamp, CurrentTemp, CurrentPressure, CurrentAltitude, CurrentLatitude, CurrentLongitude);

  if(TransmitNumber > TRANSMIT_EVERY_Nth_READING) {
    TransmitDataToGround(CurrentTimeStamp, CurrentAltitude, CurrentLatitude, CurrentLongitude);
    TransmitNumber = 0;
  } else {
    TransmitNumber++;
  }
 
  HasGoodGPSReading = false;
  HasGoodTempReading = false;
}

 

I added a new loop index called "TransmitNumber" that I would increment with every reading, and reset when I invoked TransmitDataToGround().

All appeared to be fine at first – the data was coming through exactly as I had expected.  However, I noticed that the radio wasn’t ever turning off in between transmissions.  During the nine readings that weren’t being transmitted, it would generate a solid tone.  I needed to be able to programmatically enable and disable the transmitter.  That led me to rewire the NTX2 – for full details, see the Episode 22 Addendum.

Once I had digital pin 5 wired up properly, I reworked TransmitDataToGround() to turn the transmitter at the beginning, and turn it off again at the end.

void TransmitDataToGround(float TimeStamp, float Altitude, float Latitude, float Longitude) {
 
  // Build the string to transmit
  digitalWrite(RADIO_ENABLE_PIN, HIGH);

  dtostrf(TimeStamp, 20, 3, TimeStampBuffer);
  dtostrf(Altitude, 20, 3, AltitudeBuffer);
  dtostrf(Latitude, 20, 5, LatitudeBuffer);
  dtostrf(Longitude, 20, 5, LongitudeBuffer);

  int TransmitIndex = 0;
  TransmitIndex = TransferBuffer(CALL_SIGN, sizeof(CALL_SIGN), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(TimeStampBuffer, sizeof(TimeStampBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(AltitudeBuffer, sizeof(AltitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LatitudeBuffer, sizeof(LatitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LongitudeBuffer, sizeof(LongitudeBuffer), TransmitBuffer, TransmitIndex, true);

  unsigned int CHECKSUM = gps_CRC16_checksum(TransmitBuffer);  // Calculates the checksum for this datastring
  sprintf(TransmitBufferChecksum, "*%04X\n", CHECKSUM);
  TransmitIndex = TransferBuffer(TransmitBufferChecksum, sizeof(TransmitBufferChecksum), TransmitBuffer, TransmitIndex, false);
 
  rtty_txstring (TransmitBuffer);
  digitalWrite(RADIO_ENABLE_PIN, LOW);
}

Monkey Wrench #4

I found that when I enabled the transmitter, I was losing several characters off of the beginning.  This was because dl-fldigi needed 2-3 seconds between when the transmitter booted up until it actually started sending data to find the signal again, and start decoding it.  I added a 3-second delay to TransmitDataToGround, right after I enabled the transmitter, to provide this buffer.

void TransmitDataToGround(float TimeStamp, float Altitude, float Latitude, float Longitude) {
 
  // Build the string to transmit
  digitalWrite(RADIO_ENABLE_PIN, HIGH);
  delay(3000);
  dtostrf(TimeStamp, 20, 3, TimeStampBuffer);
  dtostrf(Altitude, 20, 3, AltitudeBuffer);
  dtostrf(Latitude, 20, 5, LatitudeBuffer);
  dtostrf(Longitude, 20, 5, LongitudeBuffer);

  int TransmitIndex = 0;
  TransmitIndex = TransferBuffer(CALL_SIGN, sizeof(CALL_SIGN), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(TimeStampBuffer, sizeof(TimeStampBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(AltitudeBuffer, sizeof(AltitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LatitudeBuffer, sizeof(LatitudeBuffer), TransmitBuffer, TransmitIndex, true);
  TransmitIndex = TransferBuffer(LongitudeBuffer, sizeof(LongitudeBuffer), TransmitBuffer, TransmitIndex, true);

  unsigned int CHECKSUM = gps_CRC16_checksum(TransmitBuffer);  // Calculates the checksum for this datastring
  sprintf(TransmitBufferChecksum, "*%04X\n", CHECKSUM);
  TransmitIndex = TransferBuffer(TransmitBufferChecksum, sizeof(TransmitBufferChecksum), TransmitBuffer, TransmitIndex, false);
 
  rtty_txstring (TransmitBuffer);
  digitalWrite(RADIO_ENABLE_PIN, LOW);
}

Now when the radio turns on, the waterfall in dl-fldigi looks like this:

10

The blank horizontal line in the waterfall above shows this 3-second delay.  The red line that starts out horizontal, and then quickly curves to be vertical is the data coming over the airwaves.  That red line, and the accompanying one to the left, need to be lined up with the two thin red lines that make up the dl-fldigi cursor.  Until those two pairs of lines line up, nothing is decoded. 

I also found that the transmitter will drift slightly while it is transmitting, and that the dl-fldigi cursor will drift in between transmissions.  The 3-second delay gives me a chance to adjust the cursor if I need to when the next transmission starts.

Monkey Wrench #5

Even with the delay, I found I was still losing characters off the beginning of the transmission string.  I solved this by simply appending more than one copy of my call sign to the beginning.  As a result, the transmitted strings usually ends up looking like this now:

jf7&VZA-KD8VZA-KD8VZA,21515.00,314.28,0.0000,0.0000,*9395

Finally, I adjusted the squelch control in dl-fldigi to avoid having so much garbage between transmissions appear in the output window.  I still get some, but it’s not the constant barrage that it was when I started.

***

It feels good to be able to put this piece of the balloon to rest.  I may end refactoring the code a little, or adjusting it based on our instrument pack endurance/distance test, but otherwise I think its basically there.           

You can get the newly completed sketch from GitHub

July 8, 2014 Posted by | Science | Comments Off

Follow

Get every new post delivered to your Inbox.