More on testing Mp3 files and the boundary testing debate

  • Article

Over the past 3 days I have learned more about Mp3 file encoding and decoding than I have since the technology was introduced. I don't spend time downloading files from the Internet to burn CD's, I don't own an iPod or Mp3 player, or a digital video recorder. So, prior to this I haven't really paid attention to this technology and was quite ignorant of the various tools available and their capabilities. But, I must say it is pretty fascinating from a technology standpoint even though I am not an audiophile or videophile.

I still disagree with Pradeep's assertion regarding boundary testing and the notion of no fixed boundaries, but respect Pradeep's expertise in the area of Mp3 technology. An Aussie gentleman by the name of Dean Harding pointed out my incorrect assumption regarding bitrate encodings and explained the LAME encoder does allow a freeformat option in "expert" mode to produce a fixed bitrate in one kilo bit increments between 8 kb/s and 640 kb/s. (Thanks for serving up the pie Dean.) However, of 30+ common decoders I only discovered 4 decoders supported freeformatted Mp3 files even if the encoded bitrate is less than 320 kb/s. Only one decoder (WinAmp MAD) is capable of decoding files above 600 kb/s.

So, (other than me having to eat a big helping of humble pie) where does that leave us in the specific debate about boundary testing, and Pradeep's question "As a tester have you ever seen a boundary?" To that, I shall adamantly reply "yes" there are specifc boundary conditions in software. Some are easy to find, some are not so easy. A tester's ability to correctly identify a boundary value are heavily influenced by his/her in-depth domain and 'system' knowledge. For example, using the knowledge of Mp3 encodings I have learned over the past 3 days let's go back and review what tests I would design based on Pradeep's original description of the audio decoder that played an Mp3 file within the range of 24 kb/s to 196 kb/s.

Since 196 kb/s is not a standard Mp3 encoding supported by ISO standards let's assume the Mp3 player used either a Cdex, LAME, I3dec, or WinAmp MAD decoder. Using this as a reference, and some recently acquired domain knowledge I would design a set of initial tests using the following sample test data (files encoded with the specified criteria).

  1. 23, 24, 25 kb/s - Specified minimum value and minimum -1, and minimum +1 values to analyze relational operators used to artificially constrain the encoding range to a low end of 24 kb/s.
  2. 195, 196, 197 kb/s - Specified maximum value and maximum -1 and maximum + 1 values to anlyze to analyze relational operators used to artificially constrain the encoding range to a high end of 196 kb/s.
  3. 16 kb/s - this is the next ISO standard encoding bitrate below the specified minimum, so although the decoder does not support a file encoded at 23 kb/s (min -1 value) I would still want to check at the next lower standard value.
  4. 224 kb/s - this is the next ISO standard encoding bitrate above the specified minimum (same reason as explained above.)
  5. 32, 40, 48, 56, 64, 80, 96, 112, 128, 144 (see #6) , 160, 176, 192 kb/s - these are the typical ISO standard Mp3 encodings within the specified range, so we should assume that all these must work properly because there is a high probability of decoding files using these bitrates. Since there are not many of them test each one.
  6. 143, 144, 145 kb/s because the 144 kb/s bitrate seems to be an interesting value that "sticks out" more than others, and so I may also want to analyze the values around that particular value for any other anomalies
  7. Generate several randomly encoded files in the following ranges (between 24 kb/s and 127 kb/s) (between 128 and 143 kb/s), and (between 145 kb/s and 196 kb/s) to gain confidence the decoder can decode non-standard encoded files within the specified range without having to test all 174 or so possibilities

These are not the only tests I would execute; however, they would be the first set of tests I would design and execute to make sure the code at least does what it is supposed to do. Any failure in the above cases means our basic program functionality has some serious flaws. Once I established the program does what it is supposed to do (including handling expected errors gracefully), then I would begin exploring other possibilities including rigorous falsification/negative testing.

Pradeep indicated a file encoded at 96 and 128 kb/s crashed the system. These are not boundaries conditions (unless the developer did something totally unreasonable), and unfortunately since we can only assume files encoded above and below 96 and 128 kb/s played correctly we will never really know the cause of this problem (unless Pradeep did some root cause analysis and will share those findings). However, a failure with 128 kb/s is really a red flag to me because this happens to be the most prevalent bitrate for encoding Mp3 files. As a tester I would really want to know why unit testing or build acceptance testing, etc. didn't at least hit the most probable encoding format (the happy path) before throwing crap code over the wall for Pradeep to test.

I hope the reader takes away a few lessons from all this (besides the obvious one of not going off half cocked especially if you lack expertise in the specific context (e.g. Mp3 encodings)). For example, 

  • In-depth knowledge of the domain space (including the data set, how the data is encoded, and how the data set can and cannot be manipulated in code both correctly and incorrectly), industry standards, and how the domain space interacts with the system are critical for greater test effectiveness
  • The less we know about the domain, the data set, the system interaction the less effective are the application of specific techniques to identify specific classes of defects
  • Boundary testing is simply one technique (systematic procedure to solve one type of a complex problem). The boundary value analysis technique is designed to identify a specific class of defects involving incorrectly specified constant values, incorrect use of data types, or casting between data types, artificially constraining data types, and incorrect usage of relational operators. It is not effective to identify other classes of defects.
  • Boundary conditions simply do not exist at the extreme physical ranges, there could also be multiple boundary conditions within the overall range. (A good example is the Unicode repetoire. The Unicode BMP spans from U+0000 to U+FFFF, but within this range there are several important boundaries one must take into consideration when using Unicode data depending on the purpose of the test and the application under test (e.g. Private use area, surrogate pair area, etc).)
  • Understanding how to decompose the data set into equivalence class partition subsets exposes boundary conditions we might not otherwise consider
  • There is a great deal of detail in the code that can expose interesting information for a tester
  • When talking technology, be as specific and precise as possible to avoid ambiguity
  • And perhaps most importantly, one technique or one approach to testing is not sufficient. As testers we must gather and learn to use a great variety of skills and knowledge and approach the problem from multiple perspectives to be most effective in our roles.

OK...now time to get some really bitter coffee to wash down that humble pie :-)