creating exams, your thoughts?


Howard, developer certification planner here at MSL, is looking for feedback on how we can be more creative in exam development (and not get shut down by “good idea, but there’s no budget.”)  He also has a picture of a lake in the NW Cascades on his blog. Both are worth checking out: Http://blogs.msdn.com/howard_dierking/archive/2007/06/05/community-based-development-let-s-talk.aspx

Comments (15)

  1. Some of these are ideas for exam creations, others are ideas for the certification department in general.

    1.) Adaptive exams, or more of them.

    2.) More simulation questions. Not drag and drop but actual simulation questions.

    4.) More choices for multiple choice answers. By this I mean instead of having the standard 4 answers to choose from, make it more difficult by having 6-8 closely worded answers.

    3.) No more Beta exams (or) do not post scores and do not award certification for beta exams.

    4.) Sorry about this but: Hands-on exams. I know, I know… There is a budget.

    I am sure I have more, but thats all for now.

  2. hdierking says:

    1) the ability to do adaptive is determined by the size of the item pool (community based development opens up this option)

    2) community based development reduces cost of standard question type development – freeing budget for more innovative question types.

    4.1) this is really a question of the statistical element.  With MC questions, studies have demonstrated many times (inside/outside of MSFT) that once you get more than 4 answer choices, the question does not "perform" well.  By "perform" we are not measuring difficulty – we are measuring the ability of a question to discriminate between a qualified and non-qualified candidate.

    3) We set the passing score based on the beta.  Why do you believe that betas are a problem?

    4) right – see comment to #2

  3. Howard, shoot me an email at mdalligood@bellsouth.net

    I want to get a little more info from you regarding this. Thanks!

  4. Well, from the looks of it, I may be the only person that does not agree with what Howard has proposed.

    I believe that, over a reletively small amount of time, his proposal would cost Microsoft more money than if they were to just remain on the current track they’re on.

    I was going to respond to his proposal directly on his Blog, but it turned out to be much longer than expected. So, I have posted a detailed description of my arguments here: http://www.certguard.com/forums/topic.asp?TOPIC_ID=198

    Please let me know what you think as well.

  5. Bob G. says:

    Anything you can do to drive traffic to your site!

  6. This is in response to Howard Dierking’s Blog – Community Based Development – Let’s Talk

    I’m responding here so that I can get our readers involved in this conversation as well. I will be providing a link to this post on Howard Dierking’s Blog as well.

    I have read all the comments on Howard Blog, as well as comments on Trika’s Blog and Deepak Kapoor’s Blog

    My Thoughts

    Of all the points made about this subject, I have to agree with Deepak the most. The first of Deepak’s statements that I’d like to elaborate on is:

    quote:

    ——————————————————————————–

    Originally posted on Deepak Kapoor’s Blog

    "…creation of exam content by community must be moderated and channeled through strict review procedures. This is to ensure that candidates who will appear for exams will get the right value for money. Implementing a rigorous quality assurance process will require budgets, and this takes us back to the main issue here."

    ——————————————————————————–

    A point that I would like to add is the fact that involving a community would require years of preparation that would also cost alot of money that the current budget, obviously, doesn’t have.

    Questions have to be answered:

    1. Who will head-up the community involvement? (You can’t just let a community loose on the creation of an exam with nobody to corroborate the information) :Budget+

    2. Once the information has been collected, who will weed out the numerous duplicate/redundant/worthless/pointless questions and make sure that only the relevant questions make it into the pool? :Budget+

    3. Who is going to keep people from bickering about those numerous duplicate/redundant/worthless/pointless questions and make sure that people focus on what is relevant and not what they BELIEVE other’s should know? (SMEs KNOW what others should know) :Budget+

    4. Once a fraction of those questions have been processed into the pool, who will determine the Answers are worded so as to not create serious confusion amongst the candidates? :Budget+

    5. Now the Questions and Answers have been collected, weeded, and sorted, who will approve them? :Budget+

    Pain Point #1: The more hands that are in the pot, the harder it becomes to control the product. If you have a community of people writing questions, you’re going to have a harder time regulating that community. :Budget+

    Pain Point #2: It would cost more money in the long run to recall those (now displaced/disgruntled) SMEs and pay them more money to repair a problem that they had under control in the first place. :Budget+

    As it stands now (you say) there are 5-7 SMEs that are writing exams in specific areas where they are the experts. From what I understand, those Experts are involved in many different aspects of creating an exam. To replace them with a community of people that don’t have the knowledge that the SMEs have, you’re asking for trouble.

    It takes professionals to Plan Examinations. What you’re asking for would be similar to asking College Students what they would like to see on next week’s exam. It is simply something that cannot be done.

    Back to Howard’s Points

    Community-based question development.

    quote:

    ——————————————————————————–

    Originally Posted on Howard Dierking’s Blog

    1. Authors must be subject matter experts across the entire domain – this comes at a premium.

    2. Authors must write a lot of questions each – this can lead to author burnout.

    3. Authors must potentially write questions on topics that they are not all that passionate about.

    4. The total number of items is all we get for an exam. If somebody comes out with a dump, the exam is exposed without much recourse (other than starting the whole process over again). Additionally, any questions that we kill (for whatever reason) are simply lost – there’s no new questions to fill the spot – the result is simply a smaller question pool.

    ——————————————————————————–

    My responses

    1. (a)Good. That proves to the Candidates that the Exams will be worthy of taking, versus someone figuring that a community of misfits pulled the questions out of their….

    (b)In addition, this ‘premium’ ultimately comes at a lower cost that the potential losses that may acrue because of the overall lack of relevant knowledge of a community based system.

    2. Highly Doubtful. Possible, yes, but doubtful. Writer’s block would be a more feasible reach for an answer than Burnout. What do you think the burnout rate of 1000s of community members would be?

    3. Passionate or not, they get paid to write those questions because they know the answers before the questions are asked.

    4. Ok, I do not agree with their methods on "Sealing" an exam once it has been released and I do believe that questions should be replaced once they are removed (If not for security purposes, at least to keep the candidates honest), but I still do not agree that a community of under-qualified IT Professionals should be writing the exams. If more people want the ability to write the exams, then more people need to work on their qualifications of becoming Microsoft Certified Trainers.

    Besides, there is already a community that does this. It’s called the MCT Community.

    According to Microsoft

    quote:

    ——————————————————————————–

    Originally published on http://www.microsoft.com/learning/mcp/mct/default.mspx

    Microsoft Certified Trainers (MCTs) are the premier technical and instructional experts in Microsoft products, technologies, and solutions. To ensure a superior learning experience for our customers, MCTs are the only individuals that Microsoft authorizes to deliver training that is based on Official Microsoft Learning Products or Microsoft Dynamics Learning Products for the MCT program.

    The MCT community spans more than 140 countries and includes classroom and e-learning instructors, learning consultants, authors, conference presenters, user group leaders, and more. All are united by passion and talent for helping Microsoft customers and partners realize their full learning potential.

    MCTs meet high initial and on-going certification requirements and enjoy access to many benefits. These include free access to the complete library of Official Microsoft Learning products; substantial discounts on exams, books, and Microsoft products; members-only newsgroups and online community resources; and invitations to exclusive events and programs.

    ——————————————————————————–

    If anyone thinks they can do the job of an MCT, they need to EARN the rights to be an MCT.

    Read the MCT Program Guide AND the MCT Program Agreement. Then, follow the 4 basic steps to becoming and MCT

    On top of all the ‘question writing’ that you’re proposing, how do you propose that Microsoft handle this community? You are aware that Microsoft currently audits the MCT Community throughout the year to make sure they are in compliance with program requirements, correct? Talk about throwing a wrench in those gears.

    Summary

    MCTs (SMEs/Experts) = $$

    Community = $$$

    This community you’re proposing would not be worth the time, the money, or the effort that you’re proposing from Microsoft.

    ——————————————————————————–

    CertGuard

    HAPPY BOB???

  7. Also Bob, since you’re not aware of it, I have professionals at CertGuard that are able to contribute much more than "Anything you can do to drive traffic to your site!"

    Whatever happened to the "If you can’t think of something proper to contribute, keep your mouth shut!"?

    Sorry Trika! Love Ya!!!

  8. Bob G. says:

    KISS MY ASS bud.

  9. Bob G. says:

    I hope you learned your lesson.

  10. Reneg says:

    Who died and promoted this Robert Williams guy king? After looking at his web site CertGuard, here are my comments:

    CertGuard is like the civilian militiamen down on the Texas border trying to prevent illegal aliens from crossing over and like those militiamen is overstepping its bounds. Having a chip on your shoulder for the test takers cheating on exams and trying to "educate them" on their sinful ways is going to have as much success as trying to keep teenagers from having sex by promoting abstinence or trying to keep people from smoking marijuana by making it an illegal drug. The only ones who are going to fix this problem are the certification providers like Microsoft who rather than keep offering their weak multiple-choice based tests or even the simulation-based questions that are easily brain dumped, to instead implement hands-on-certifications like Cisco’s CCIE. How many paper CCIEs are there in the world? Answer: None.

    So if you want to fix the exam proces – CHANGE THE NATURE OF THE EXAM!

  11. trikah@microsoft.com says:

    ‘KISS MY ASS bud’ is going to be my new tag line. I love it!

    How many CCIEs are there (Per Reneg, paper CCIE = zero; non-paper CCIE = ?). Anyone know off the top of their head?

  12. Bob G. says:

    Actually I was thinking of changing that to DUMB ASS, as Red would say.

    ~14,387 CCIEs around the world;

    http://www.cisco.com/web/learning/le3/ccie/certified_ccies/worldwide.html

  13. CertGuard says:

    ReneG (if that’s your real name…*cough* Scooby), you can look at it however you want. The point of CertGuard is to take the focus off of the Braindump websites and provide candidates with better alternatives to studying.

    Bob G., you’ve stated that there are ZERO paper CCIEs out of ~14,387 in the world. After reading a little, you may be right about that zero, but lets look at the REAL reasons there are very few CCIE certificates out there.

    #1 $300 USD for the written exam

    #2 $1,250 USD for the lab exam (per attempt)

    #3 Exams offered in 10 locations (worldwide)

    #4 Average expenses of preparation: $9,000+ (USD)

    #5 The average annual income in India and China is about $500 USD.

    Now, that pretty much rules out India and China unless the person is rich or has rich family. Ok, no gunmen will touch it. Good.

    Now, I’m sure that of the people in the other countries, maybe a few percent are even interested, and of those, even fewer percent can afford it. I know that I’ve been fairly well off my entire life and I wouldn’t spend the money on that.

    It’s not that I couldn’t, because I know that if I applied myself to it, I could have that CCIE within the allotted time frame…and I’m sure I’m not the only one that refuses to pay that much for a certification.

    Whooo Hoooo…."CertGuard, the Militiamen of IT Certifications!"

    "Kiss my shiny metal ass!!" ~bender, Futurama

  14. Bob G. says:

    There’s no "paying" for certifications. It’s a privilege and not a right. Knowledge is power!

Skip to main content