More on boundary testing and Mp3 encodings


My previous post refuting a conjecture by Pradeep Soundarajan suggesting there are no boundary values in software was a bit harshly worded, and to him and the readers I apologize. Occasionally I get a little overzealous. I am sure Pradeep is a great guy, and I must say his reply to me on his blog was rather cordial given the situation. As I told Pradeep, email and blogs are a poor medium for expressing emotion. MichaelB cautioned me about this before, but my Type A personality sometimes takes over, and it is something I need to work on.


 Anyway, although my critique of Pradeep's conjecture was pretty ruthless, the analysis was accurate and the  boundary values Pradeep suggested for testing an Mp3 file are in fact not possible or probable even using the tools he referenced. I am not an expert on Mp3 encodings or decoding technology, but i did know that Mp3 files used standard encoding bit rate formats. A little investigation quickly revealed the bit rates for audio encoding are based on multiples of 8, and the first 32-bits of an Mp3 file contain header information including 4 bits to specify the bit rate index (Layer 1, Layer 2, or Layer 3) and the bit rates as outlined below. (Thanks to Wkikpedia some of the specific data I used in my initial rebuttal was incorrect, and I put a single strikethrough that part of the sentence.)



Layer 1 - 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448
Layer 2 - 32, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320, 384
Layer 3 - 32, 40, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320


In his attempt to contest my argument Pradeep said, "Pooh! I am not sure why you don't know that bit rates can be 33, 41, 57 or any number you want to generate. I recommend you to go through some multimedia test content generation tools like ffmpeg which gives a multimedia tester an edge to generate test content of his choice."


So, not being an expert I took Pradeep's suggestion "For those who don't know tools like ffmpeg, it is impossible. I suggest you explore the boundary of your education on multimedia." and went home and increased my understanding of Mp3 encodings and the ffmpeg toolset. As I read through some of the API references I found an interesting struct (illustrated below) for bit rate constants. (Now, I am thinking to myself...this is a clue! I am also thinking...hmmm...there might be some real boundary values here!)



static const int sBitRates[2][3][15] = {
00085     {   {  0, 32, 64, 96,128,160,192,224,256,288,320,352,384,416,448},
00086         {  0, 32, 48, 56, 64, 80, 96,112,128,160,192,224,256,320,384},
00087         {  0, 32, 40, 48, 56, 64, 80, 96,112,128,160,192,224,256,320}
00088     },
00089     {   {  0, 32, 48, 56, 64, 80, 96,112,128,144,160,176,192,224,256},
00090         {  0,  8, 16, 24, 32, 40, 48, 56, 64, 80, 96,112,128,144,160},
00091         {  0,  8, 16, 24, 32, 40, 48, 56, 64, 80, 96,112,128,144,160}
00092     },


I also followed up by asking a co-worker who frequently works with ffmpeg to try to encode a Mp3 file with a bit rate of 57 kb/s. Interestingly enough, when he issued the command line parameters we got an error message indicating "Invalid Value." (I can get a snapshot of the command window, but I really don't think that is necessary.)


I am still not an expert on Mp3 encodings, but I am fairly certain that Mp3 file decoding algorithms are standardized across the industry. So, let's just assume for a moment that we can encode a Mp3 file at 57 kb/s, and that file fails to play. Does it really matter? No, because industry hardware simply doesn't support that encoding, and as long as the Mp3 player didn't burst into flames there is no business case that would compel someone to try to make it work (at this time)? (I am not suggesting that we only test only "real-world" scenarios here, but I am suggesting that in-depth domain and system knowledge goes a long way in increasing the efficiency and effectiveness of our testing (and can lead to better identification boundary values)).


Now, perhaps I am still missing something, and as I expressed previously I am not an expert on Mp3 encodings, or the use of ffmpeg or other tools to encode Mp3 files. So, I have asked Pradeep to share his knowledge with me in this area and teach me how to encode an Mp3 file with a bit rate of 57 kb/s using a commonly used tool such as ffmpeg (there is no doubt someone can write a customized algorithm to do this), and to also let me know of a commercially available Mp3 player that will decode and play that file. (Because if it can be done I would like to learn how simply because I love to learn new things.)


Many people do assume that boundary testing is quite simple. The actual execution of boundary tests are in fact rather simple; however, discerning the boundary values in any complex software is not as simple as looking at some minimum and maximum values and trying one value below and above each boundary condition. Boundary testing is a systematic procedure to solve a specific type of complex problem (specifically the incorrect usage of data types or constant values, artifically constrained data types, and relational operators). Boundary value analysis doesn't solve all problems, it is not the holy grail, and its efficacy relies on the testers ability to understand and decompose the data set effectively.The less the tester knows about the data and how the data is used by the program, the less effective they will be in the application of this technique.


I did not intend my previous post to be construed as a personal attack against Pradeep; I am sure he is a bright guy. But, I am challenging his assertion on boundary testing on its technical merit. I hope he replies here (or on his blog) with an example of how to encode an Mp3 file at 57 kb/s, and I will make sure it is posted (or linked) here because I am certainly curious. (I don't really like the taste of humble pie, but I will eat it from time to time if it helps me learn.)

Comments (7)

  1. Dean Harding says:

    So called "free format" mp3 files ARE possible (the LAME encoder is the only one I know to support it), which means you can technically create mp3 files that are any integer bitrate (up to a maximum of 640kb/s which is essentially uncompressed 44kHz audio).

    However, according to [1], there’s really only ONE decoder which actually supports all the free format bitrates.

    I do tend to agree with you, though. It’s not like boundary testing is the ONLY testing that you do, so saying "boundary testing would not find bugs in the middle of the range" displays a misunderstanding of what boundary testing is FOR, imo.

    [1] http://mp3decoders.mp3-tech.org/freeformat.html

  2. Shrini says:

    Good Post BJ,

    Happy to see that you seem to have changed few of your "hard core" beliefs about boundary Testing ….At the end we all learn.

    Now — What do you say about upper boundary for a 32 bit unsigned integer field – is this fixed? or can be tampered with? are there multiple boundaries?

    I got this from your earlier post where you referenced few of the well known boundaries like 65535 etc.

    Do you think there are certain boundaries that are "universal" and hence nearly "fixed" ?

    Shrini

  3. I.M.Testy says:

    Hi Shrini,

    Actually, I don’t have any ‘beliefs’ about boundary testing. Boundary testing is an analytical process based on rational thought. But, no, I have not changed my position on the application or value of using boundary value analysis as a technique to identify specific classes of defects. The more we know, the greater its effectiveness. As I said, I know that boundary testing is not a panacea for everything, but it is a very useful tool, and is very good at doing the job it was designed to do.

    (I really hate to repeat myself, but sometimes people only read what they want to read, or misinterpret the words I use.) So, I have said all along, there are often multiple boundaries between the physical ranges of primitive data types and I specifically , and they are sometimes very hard to find withuot looking at the code or in-depth domain and system level knowledge. I specifically said “Also, if I artificially constrain an int in a predicate statement using a relational operator such as if (intValue <= 0) then there is another boundary condition that I would certainly want to analyze” and “testers must be aware that boundary values don’t always exist only at the extreme ranges of data types or other variables. Occasionally, there are boundary values/conditions within the minimum and maximum physical ranges of a variable.” I am trying to pass on some useful information here, so if any of this is unclear please let me know.

    But, perhaps I am missing the intent of your question, or the point you are tyring to make (and I sure hope that point is not suggesting the upper boundary for a 32 bit unsigned int can be greater than 4294967295). If so, can you please rephrase your question concisely?

  4. Shrini says:

    Hi BJ,

    My question was more about “very well known” boundaries and “not so well known or unknown” boundaries.

    I believe there are some boundaries that can be reasonably considered to  be “clear” and known – like “32 bit unsigned int can be greater than 4294967295” (Let me know if I am wrong)

    [Bj] I assume you mean a 32 bit unsigned int cannot be greater than 4294967295.

    So can you please elaborate on approach to identifying “known” and “unknown boundaries”.

    If we agree that given a test object with several variables – there are some known boundaries and some unknown. Hence, Pradeep’s assertion that “he has not seen boundary” seems correct (at least with respect to unknowns)

    [Bj] I take his assertion to imply that there are no defined boundary conditions in software. Even if the boundaries are unknown, testers should assume that any time there are physical values in play there are definate boundary values that must be considered (unless of course we can prove otherwise). But, it is probably foolish for me at this point to make any assumptions as to his implied meaning.

    Coming back to your differentiation between “Extreme ranges of data types and other variables (I am not sure what other variables you are referring to)” vs. “Maximum and minimum physical ranges of variables” – Can you help me with an example? This can be an interesting twist to our discussion on boundary values.

    [Bj] The use of the word variable is often overloaded. But, perhaps a simple example is the extreme range of an integral uint type in C# is 0 to 4,294,967,295. But, if the program only allowed input values greater than or equal to 1 and less than or equal to 10, then the minimum and maximum physical range of the user input (variable) is artifically constrained between 1 and 10 although the extreme range of the type is much larger. In the example below we declare a variable of type uint which has a extreme range between 0 and 4,294,967,295. But, the conditional statement limits (or artifically constrains) the userinput assigned to the variable named input to the minimum value of 1 and maximum value of 10.

    //assuming user input is a valid integer value between 0 and 4,294,967,295
    uint input = Convert.ToUnit32(userinput);
    if ((input < 1) || (input > 10))
    {
        //sorry..invalid input
    }

    Another thing for which I am always puzzled is – about the name “Boundary Value Analysis”.

    Consider following traditional approach that I mostly heard from people

    1. Identify the variable(s)

    2. Get the boundaries of the variable from specification and by asking the developer.

    3. Create a set of values (3 per each boundary – one bellow, one on the boundary and one above it) for each variable

    4. Check the application behavior for these values

    5. Declare that Boundary testing has been performed.

    If the whole process were so simple – why it is called as “analysis” – what analysis goes here?

    [Bj] Actually, step 1 is identifying parameters that accept physical or quantifiable values (basically something that can be counted). Step 2 the physical range of allowable values can come from the spec, the dev, or the code.

    The minimum and maximum values are the specific boundary conditions or values themselves. So, one way to view analysis is the process of analyzing the value immediately above that value and immediately below the specific boundary value (per data type). Another way to view analysis is the fact that we must decompose and analyze the range and type of data for potential boundaries within the allowable range and identify unique or special values that are or could be more interesting than other values within the allowable minimum and maximum range. But, this may mostly be an academic debate as the technique has different names depending on your reference.

    What are your experiences?

    [Bj] In my experience, most people have a limited understanding of boundary testing from a very superficial point of view which is often based only on the minimum and maximum input and output values (although most people also forget about output values). The most common problem is the failure to decompose the data to identify additional boundary values or potential boundary conditions that could exist within the allowable (and most easily identified) physical range. Of course, the ability and skill to decompose data takes a great deal of time, expertise, and in-depth knowledge of the data set, the programming language in use, the algorithm, the system, etc. (This is one example of the types of abilities and skills we are expecting from our highly skilled testers at Microsoft and our training is designed to teach and reinforce this and other skills necessary to become more effective and more efficient.)

    The mere execution of testing boundaries is pretty much rudimentary as you describe above once the boundary conditions or values are identified. The real skill in the application of the technique of boundary value analysis (or boundary testing) is not in the execution of the test, but in the ability to identify specific and probable boundary conditons (especially those within the obvious minimum and maximum ranges).

    Shrini

  5. I.M.Testy says:

    Shrini,

    These are really great questions! I replied inline above in blue. Please let me know if there is something I did not clarify sufficiently.

    – Bj –

  6. Shrini says:

    Thanks BJ for the clarifications. It really helped. I will come back with more questions in due course.

    [Bj] I assume you mean a 32 bit unsigned int cannot be greater than 4294967295.

    [shrini] Yes that was my assumption. Is this boundary fixed or not. If I overflow this with some value which is greater and managed to go past it and hit another boundary say at 5000000000 – will this remain as fixed? Here tester’s curiosity or hunch that "things can be different" is at play and might discover few interesting things. What do you say?

    I have a blog post for you see and comment at

    http://shrinik.blogspot.com/2007/03/boundary-value-exploration-bve.html

    I am calling the whole approach of boundary testing as "Boundary Value Exploration".

    >>[Bj] Actually, step 1 is identifying parameters that accept physical or quantifiable values (basically something that can be counted).

    [shrini] That is BVE -Outsider’s view for me.

    >>> [Bj] Step 2 the physical range of allowable values can come from the spec, the dev, or the code.

    [shrini] That is BVE – Insider’s view.

    >> [BJ]The minimum and maximum values are the specific boundary conditions or values themselves. So, one way to view analysis is the process of analyzing the value immediately above that value and immediately below the specific boundary value (per data type).

    [Shrini] Again, BVE -Outsider’s view

    >> [BJ]Another way to view analysis is the fact that we must decompose and analyze the range and type of data for potential boundaries within the allowable range and identify unique or special values that are or could be more interesting than other values within the allowable minimum and maximum range.

    [Shrini] BVE – Insider’s view.

    Thanks again

    Shrini

  7. I.M.Testy says:

    It is mathematically impossible for 2 raised to the 32nd power to be greater than 4294967295 (4294967296 – 1 because we start counting at zero).

    Therefore, if the developer declares a variable with an unsigned 32-bit integer type (which is an integral data type) and the user enters a value > 4294967295 then an overflow exception will be thrown (and hopefully the developer has error code to deal with that situation.

    Now, if you can enter a value of 5,000,000,000 and the application does not throw an exception then the developer has not declared the variable as a 32-bit unsigned integer, but (mostly likely) as a signed or unsigned 64-bit integer data type.

    If the developer chooses to artificially constrain that 64-bit integer to 5000000000 based on some requirements, then it is most likely that will remain fixed because I don’t know of too many developers who arbitrarily change an established boundary value in the code very frequently. In fact, in well-formed code the value used to contrain the size of an integral data type is typically declared as constant value and not hard-coded into a conditional statement.

    Curiosity is good, and curiosity often helps us discover new and interesting things. However, between 4294967295 (max value for unsigned 32 bit integer) and 9,223,372,036,854,775,807 (max value for a signed 64 bit integer) there are 9,223,372,032,559,808,512 integer values. Personally, I don’t want to rely on hunches or best guess to see if the developer has artifically constrained variable declared as a 64 bit integer. I will find the specific value from the requirements, code, developer, etc. and take it for a boudnary test ride. Once I do that, if I want to poke around at random values to see if anything interesting happens, then that my friend is called equivalence class testing.

    Karen Johnson gives some pretty good examples of specific boundary values in an earlier post. She states, she doesn’t think about the numbers technically, but the fact is there is a technical basis why some of the numbers she refers to are extremely valuable to know and why they are more interesting than other numbers.

    For example, she discusses the value 2038 in the context of date input. The link in her post does an excellent job of explaining the details of the problem. However, on Jan 19, 2038 at precisely 03:14:07 the second count will reach the max size of a signed 32-bit integer value and the second count will wrap around to -2147483648. Not only is this particular issue interesting in date fields (and in fact has mostly been corrected), but any place in the program which uses time as an operand to perform internal calculations should be suspect.

Skip to main content