Merging coverage reports with Bullseye


I’ve previously recommended Bullseye. And there is another nifty feature with Bullseye you should know; the ability to merge reports. This is pretty useful when you have one report from your unit tests and one from some other type of test run. Use this command to merge reports with Bullseye:

covmerge.exe -c -fMergedData.cov file1.cov file2.cov

Comments (22)

  1. Abhilash says:

    I have merged 2 .cov files, how to identify that the generated .cov file is correctly merged.

    Thanks

  2. Why would you like to verify this? Why not trust the tool? Why would it not be correct?

    But the way to verify is to look at the data and the merged data will be max coverage in the two original files. Let’s assume that your first coverage file tells you that function A has 25% coverage and function B has 50% coverage. The second coverage file then has function B at 75% and function C at 100% coverage. Then the merged file should show: A-25%, B-75%, C-100%.

  3. ram says:

    re: Merging coverage reports with Bullseye

    >Then the merged file should show: A-25%, B-75%, C-100%.

    This doesnt always happen. If function B has modified, bullseye retains the cov for the one which is more recent.

    So if the second coverage file has an modified function B , and has coverage B- 25% and Func C at 100%, then the merged file would show

    A-25%, B-25% , C-100%

    Is there anyway to get around this?

  4. @ram:

    I didn’t know that because I run my coverage with the same build. Does not make sense to me to run coverage runs with different versions of the application. The only thing that makes sense is that if a function is changed you get the latest coverage rather than the merged because whatever value you had before is no longer valid since you changed the function. Hence I don’t think there is anything that needs to be fixed…

  5. ram says:

    >>Hence I don’t think there is anything that needs to be fixed…

    True. But consider a case where only function B is modified, (with A and C remain unchanged) and the file has a newer version. In that case merged coverage report would only be the one which corresponds to the recent version of the source.

    In which case the merged coverage would have

    A-0%, B-25% , C-100%

    and earlier obtained coverage for A is discarded. There are pros and cons for this approach, but one would expect Bullseye to leave it to the user to a pass on an parameter to covmerge to do either.

  6. @ram: I disagree that it is up to the user to decide what to do if two merge files being merged have been using different versions of the code because if there are different versions basically all coverage from the old merg file is no longer up to date. I think it is much more productive to focus on getting your different runs using the same version and doing it automatically. It just makes more sense to me.

  7. Rajan says:

    Guys, how about merging 2 coverage files build with different path. For ex: covfile1.cov has project directory c:x and covfile2.cov has project directory c:xyz. In this case whether Bullseye merger will merge the files(if the same function is not modifed; instead only new files are added in the recent covfile2.cov)?

  8. @Rajan: Never tried that. Don’t see why it should not work since the two sets of code are different. But also, since the two sets are completely different there is no benefit from joining the files other than the ability to look in one file instead of two…

  9. Rajan says:

    Let’s take one example. If more than one developer started working on one feature then as per the BullsEye’s behavior, they both have to MAINTAIN the same directory structure of their project(in their dev machines), in order to merge their .cov files from Unit Testing.

    Am I right? My question here is, why BullsEye do this much check and why don’t it identify the project/file/func names and do the merge of .cov files?

  10. Two developers working on the same project/feature should have the same folde structure because they both chech their code into some version controlsysyem, right? The root may be different, but from the rot down it should be the same. Don’t even understand why you would run coverage seperatly since there is a merged version of the code; in the version control system…

  11. Rajan says:

    Thing is, our directory structure is same but root is different. When root is different, BullsEye merge is not merging the .cov files at all.

    Fine. What i want to convey is, let’s take, Dev1 writes Func1 in File.cpp and Dev2 writes Func2 in File.cpp, both took code from same SCM. Here, Dev1 could run unit test cases to check the coverage of his func and same with Dev2. Once they are done, their manager wants to know the code coverage % of the new module. Obviously they got to merge to show the result? this is what my use case is.

    Whether they take code from SCM or not, as per the BullsEye, they have to maintain the same directory structure(from root) to build the code for code coverage and to create .cov file. when the result is asked they got to merge the .cov files.

  12. @Rajan:

    They don’t obviously have to merge their seperate results. I would say you get the latest from SCM and run the tests and get a single coverage file. I think that is the only thing that makes sense.

    I know having different roots is not a problem. Other than that I donät understand if you’ve experienced a problem and something does not work for you, in which case you need to talk to Bullseye support. Or you’re asking if this will work in which case I think there is something missing since the type of testing you describe; testing one module using two different versions does not make any sense to me.

  13. Mithun says:

    how to get separate coverage for each solution files separately.

    Example : A.sln -> coveraheA.cov

                     B.sln -> coveraheB.cov

  14. @Mithun: Before running any tests, copy the coverage file generated when compiling to the new desired names. Then set the COVFILE environment variable to different values for each solution. Or just build and run each solution seperatly deleteing (or probably moving it away for later analysis) the coverage file in between

  15. Mithun says:

    Thanks for your reply cellfish.

    I need one more help. When I'm using covc in command line with vcbuild. I get  exception like this

    Example:

    covc -fC:ProjectTestCoveragetest.cov vcbuild C:ProjectA.sln

    "Exception: unknown compiler c:/Program Files/Microsoft Visual Studio 9.0/VC/vcpackages/vcbuild.exe"

    Waiting for Ur reply

    Thanks in advance.

  16. @Mithun; I would guess that you do not run from a VS prompt but I would suggest you contact Bullseye support.

  17. Asheesh Goyal says:

    We have number of .so files. some of them are compiled with bullseyes and some are not.  how can i know which .so is linked with bullseyes.

  18. @Asheesh Goyal: Getting off topic here but assuming you're on some unix flavor ldd is the command to use: man ldd for how to use it.

  19. Giri says:

    I am facing while building driver after enabling bullseye. Build is completing but *.cov file is not generating. Any suggestion will he very helpful?

  20. @Giri It sounds like Bullseye was not enabled before you built. Or you're looking in the wrong place. I fi remember corectly the COV file is placed where it needs to be during execution which is not necesarily your build output folder. For mor ehelp I would ask the Bullseye support: bullseye.com/support.html

  21. DesireMore says:

    hi

    i wanted to find a way out to get different .cov files respective of the files that are run. If I run a batch file, then is there any method to get .cov files for every file that is run in that batch file?

    Please reply as soon as possible, Im stuck on this!

  22. @DesireMore If I remember correctly the cov file name is defined when you instrument your binary (i.e. essentially at compile time). As such you can easily give different names for each of your binaries if you want to. That is the right thing to do. A dirty work around is to rename the cove files after each run in your batch file.