The PDC and Application Compatibility, but still no Hosting

The PDC has happened, which means two things.  I
can post some of my (slightly self-censored) reactions to the show, and I can talk
about what we ve disclosed about Whidbey and Longhorn more freely.  In
this particular case, I had promised to talk about the deep changes we re making
in Whidbey to allow you to host the CLR in your process.  As
you ll see, I got side tracked and ended up discussing Application Compatibility


But first, my impressions of the PDC:


The first keynote, with Bill, Jim
& Longhorn, was guaranteed to be good.  It had all the coolness of Avalon,
WinFS and Indigo, so of course it was impressive.  In fact, throughout all the
sessions I attended, I was surprised by the apparent polish
and maturity of Longhorn.  In my opinion, Avalon looked like it is the most mature
and settled.  Indigo also looked surprisingly real.  WinFS looked good in
the keynote, where it was all about the justification for the technology.  But
in the drill-down sessions, I had the sense that it s not as far along as the others.


Hopefully all the attendees realize
that Longhorn is still a long way off.  It
s hard to see from the demos, but a lot of fundamental design issues and huge missing
pieces remain.


Incidentally, I still can t believe
that we picked WinFX to describe the extended managed frameworks and WinFS to describe
the new storage system.  One of those
names has got to go.


I was worried that the Whidbey keynote
on Tuesday would appear mundane and old-fashioned by comparison.  But to an audience
of developers, Eric’s keynote looked very good indeed.  Visual Studio looked
better than I’ve ever seen it.  The device app was so easy to write that I feel
I could build a FedEx-style package tracking application in a weekend. 


of this keynote was ASP.NET.  I hadn’t been paying attention to what they’ve
done recently, so I was blown away by the personalization system and by the user-customizable
web pages.  If I had seen a site like that, I would have assumed the author spent
weeks getting it to work properly.  It
s hard to believe this can all be done with drag-and-drop.


In V1, ASP.NET hit a home run by focusing
like a laser beam on the developer experience.  Everyone put so much effort into
building apps, questioning why each step was necessary, and refining the process. 
It’s great to see that they continue to follow that same discipline.  In the
drill-down sessions, over and over again I saw that focus resulting in a near perfect
experience for developers.  There are
some other teams, like Avalon, that seem to have a similar religion and are obtaining
similar results.  (Though Avalon desperately
needs some tools support.  Notepad is
fine for authoring XAML in demos, but I wouldn t want to build a real application
this way).


Compared to ASP.NET, some other teams
at Microsoft are still living in the Stone Age.  Those
teams are still on a traditional cycle of building features, waiting for customers
to build applications with those features, and then incorporating any feedback.  Beta
is way too late to find out that the programming model is clumsy.  We
shouldn t be shirking our design responsibilities like this.


Anyway, the 3rd keynote (from Rick
Rashid & Microsoft Research) should have pulled it all together.  I think
the clear message should have been something like:


is coming next and has great developer features.  After that, Longhorn will arrive
and will change everything.  Fortunately, Microsoft Research is looking 10+ years
out, so you can be sure we will increasingly drive the whole industry.


This should have been an easy story
to tell.  The fact is that MSR is a world class research institution.  Browse
the Projects, Topics or People categories at and
you ll see many name brand researchers like Butler Lampson and Jim Gray.  You
will see tremendous breadth on the areas under research, from pure math and algorithms
to speech, graphics and natural language.  There
are even some esoterica like nanotech and quantum computing.  We
should have used the number of published papers and other measurements to compare
MSR with other research groups in the software industry, and with major research universities.  And
then we should have shown some whiz-bang demos of about 2 minutes each.


Unfortunately, I think instead we
sent a message that Interesting technology comes from Microsoft product groups,
while MSR is largely irrelevant.   Yet
nothing could be further from the truth.  Even
if I restrict consideration to the CLR, MSR has had a big impact.  Generics
is one of the biggest feature added to the CLR, C# or the base Frameworks in Whidbey.  This
feature was added to the CLR by MSR team members, who now know at least as much about
our code base as we do.  All the CLR
s plans for significantly improved code quality and portable compilers depend on a
joint venture between MSR and the compiler teams.  To
my knowledge, MSR has used the CLR to experiment with fun things like transparent
distribution, reorganizing objects based on locality, techniques for avoiding security
stack crawls, interesting approaches to concurrency, and more.  SPOT
(Smart Object Personal Technology) is a wonderful example of what MSR has done with
the CLR s basic IL and metadata design, eventually leading to a very cool product.


In my opinion, Microsoft Research
strikes a great balance between long term speculative experimentation and medium term
product-oriented improvements.  I wish
this had come across better at the PDC.




In the 6+ years I ve been at Microsoft,
we ve had 4 PDCs.  This is the first
one I ve actually attended, because I usually have overdue work items or too many
bugs.  (I ve missed all 6 of our mandatory
company meetings for the same reason).  So
I really don t have a basis for comparison.


I guess I had expected to be beaten
up about all the security issues of the last year, like Slammer and Blaster. 
And I had expected developers to be interested in all aspects of security.  Instead,
the only times the topic came up in my discussions is when I raised it.


However, some of my co-workers did
see a distinct change in the level of interest in security.  For
example, Sebastian Lange and Ivan Medvedev gave a talk on managed security to an audience
of 700-800.  They reported a real upswing
in awareness and knowledge on the part of all PDC attendees.


But consider a talk I attended on
Application Compatibility.  At a time
when most talks were overflowing into the hallways, this talk filled less than 50
seats of a 500 to 1000 seat meeting room.  I
know that AppCompat is critically important to IT.  And
it s a source of friction for the entire industry, since everyone is reluctant to
upgrade for fear of breaking something.  But
for most developers this is all so boring compared to the cool visual effects we can
achieve with a few lines of XAML.


Despite a trend to increased interest
in security on the part of developers, I suspect that security remains more of an
IT operations concern than it does a developer concern.  And
although the events of the last year or two have got more developers excited about
security (including me!), I doubt that we will ever get developers excited about more
mundane topics like versioning, admin or compatibility.  This
latter stuff is dead boring.


That doesn t mean that the industry
is doomed.  Instead, it means that modern
applications must obtain strong versioning, compatibility and security guarantees
by default rather than through deep developer involvement.  Fortunately,
this is entirely in keeping with our long term goals for managed code.


With the first release of the CLR,
the guarantees for managed applications were quite limited.  We
guaranteed memory safety through an accurate garbage collector, type safety through
verification, binding safety through strong names, and security through CAS.  (However,
I think we would all agree that our current support for CAS still involves far too
much developer effort and not enough automated guarantees.  Our
security team has some great long-term ideas for addressing this.)


More importantly, we expressed programs
through metadata and IL, so that we could expand the benefits of reasoning about these
programs over time.  And we provided metadata
extensibility in the form of Custom Attributes and Custom Signature Modifiers, so
that others could add to the capabilities of the managed environment without depending
on the CLR team s schedule.


FxCop (
is an obvious example of how we can benefit from this ability to reason about programs.  All
teams developing managed code at Microsoft are religious about incorporating this
tool into their build process.  And since
FxCop supports adding custom rules, we have added a large number of Microsoft-specific
or product-specific checks.



Churn and Application Breakage

We also have some internal tools that
allow us to compare different versions of assemblies so we can discover inadvertent
breaking changes.  Frankly, these tools
are still maturing.  Even in the


timeframe, they did a good job of blatant violations like the removal of a public
method from a class or addition of a method to an interface.  But
they didn t catch changes in serialization format, or changes to representation after
marshaling through PInvoke or COM Interop.  As
a result, we shipped some unintentional breaking changes in


, and until recently we were on a path to do so again in Whidbey.


As far as I know, these tools still
don t track changes to CAS constructs, internal dependency graphs, thread-safety
expectations, exception flow (including a static replacement for the checked exceptions
feature), reliability contracts, or other aspects of execution.  Some
of these checks will probably be added over time, perhaps by adding additional metadata
to assemblies to reveal the developer s intentions and to make automated validation
more tractable.  Other checks seem like
research projects or are more appropriate for dynamic tools rather than static tools.  It
s very encouraging to see teams inside and outside of Microsoft working on this.


I expect that all developers will
eventually have access to these or similar tools from Microsoft or 3rd parties,
which can be incorporated into our build processes the way FxCop has been.


Sometimes applications break when
their dependencies are upgraded to new versions.  The
classic example of this is Win95 applications which broke when the operating system
was upgraded to WinXP.  Sometimes this
is because the new versions have made breaking changes to APIs.  But
sometimes it s because things are just different .  The
classic case here is where a test case runs perfectly on a developer s machine, but
fails intermittently in the test lab or out in the field.  The
difference in environment might be obvious, like a single processor box vs. an 8-way.  Yet
all too often it s something truly subtle, like a DLL relocating when it misses its
preferred address, or the order of DllMain notifications on a DLL_THREAD_ATTACH.  In
those cases, the change in environment is not the culprit.  Instead,
the environmental change has finally revealed an underlying bug or fragility in the
application that may have been lying dormant for years.


The managed environment eliminates
a number of common fragilities, like the double-free of memory blocks or the use of
a file handle or Event that has already been closed.  But
it certainly doesn t guarantee that a multi-threaded program which appears to run
correctly on a single processor will also execute without race conditions on a 32-way
NUMA box.  The author of the program must
use techniques like code reviews, proof tools and stress testing to ensure that his
code is thread-safe.


The situation that worries me the most is when an application
relies on accidents of current FX and CLR implementations.  These
dependencies can be exceedingly subtle.


Here are some examples of breakage that we have encountered,
listed in the random order they occur to me:


  1. Between V1.1 and Whidbey, the implementation of reflection
    has undergone a major overhaul to improve access times and memory footprint.  One
    consequence is that the order of members returned from APIs like Type.GetMethods has
    changed.  The old order was never documented
    or guaranteed, but we ve found programs including our own tests which assumed
    stability here.


  1. Structs and classes can specify Sequential, Explicit
    or AutoLayout.  In the case of AutoLayout,
    the CLR is free to place members in any order it chooses.  Except
    for alignment packing and the way we chunk our GC references, our layout here is currently
    quite predictable.  But in the future
    we hope to use access patterns to guide our layout for increased locality.  Any
    applications that predict the layout of AutoLayout structs and classes via unsafe
    coding techniques are at risk if we pursue that optimization.


  1. Today, finalization occurs on a single Finalizer thread.  For
    scalability and robustness reasons, this is likely to change at some point.  Also,
    the GC already perturbs the order of finalization.  For
    instance, a collection can cause a generation boundary to intervene between two instances
    that are normally allocated consecutively.  Within
    a given process run, there will likely be some variation in finalization sequence.  But
    for two objects that are allocated consecutively by a single thread, there is a high
    likelihood of predictable ordering.  And
    we all know how easy it is to make assumptions about this sort of thing in our code.


  1. In an earlier blog (,
    I talked about some of the circumstances that impact when the JIT will stop reporting
    a reference to the GC.  These include
    inlining decisions, register allocation, and obvious differences like X86 vs. AMD64
    vs. IA64.  Clearly we want the freedom
    to chase better code quality with JIT compilers and NGEN compilers in ways that will
    substantially change these factors.  Just
    yesterday an internal team reported a GC bug on multi-processor machines only
    that we quickly traced to confusion over lifetime rules and bad practice in the application.  One
    finalizable object was accessing some state in another finalizable object, in the
    expectation that the first object was live because it was the this argument
    of an active method call.


  1. During V1.1 Beta testing, a customer complained about
    an application we had broken.  This application
    contained unmanaged code that reached back into its caller s stack to retrieve a
    GCHandle value at an offset that had been empirically discovered.  The
    unmanaged code then transitioned into managed and redeemed the supposed handle value
    for the object it referenced.  This usually
    worked, though it was clearly dependent on filthy implementation details.  Unfortunately,
    the System.EnterpriseServices pathways leading to the unmanaged application were somewhat
    variable.  Under certain circumstances,
    the stack was not what the unmanaged code predicted.  In
    V1, the value at the predicted spot was always a 0 and the redemption attempt failed
    cleanly.  In V1.1, the value at that stack
    location was an unrelated garbage value.  The
    consequence was a crash inside mscorwks.dll and Fail Fast termination of the process.


  1. In V1 and V1.1, Object.GetHashCode() can be used to obtain
    a hashcode for any object.  However, our
    implementation happened to return values which tended to be small ascending integers.  Furthermore,
    these values happened to be unique across all reachable instances that were hashed
    in this manner.  In other words, these
    values were really object identifiers or OIDs.  Unfortunately,
    this implementation was a scalability killer for server applications running on multi-processor
    boxes.  So in Whidbey Object.GetHashCode()
    is now all we ever promised it would be: an integer with reasonable distribution but
    no uniqueness guarantees.  It s a great
    value for use in HashTables, but it s sure to disappoint some existing managed applications
    that relied on uniqueness.


  1. In V1 and V1.1, all string literals are Interned as described
    in  I
    noted there that it is a mistake to depend on Interning across assemblies.  That
    s because the other assembly might start to compose a String value which it originally
    specified as a literal.  In Whidbey, assemblies
    can opt-in or opt-out of our Interning behavior.  This
    new freedom is motivated by a desire to support faster loading of assemblies (particularly
    assemblies that have been NGEN ed).  We
    ve seen some tests fail as a result.


  1. I ve seen some external developers use a very fragile
    technique based on their examination of Rotor sources.  They
    navigate through one of System.Threading.Thread s private fields (DONT_USE_InternalThread)
    to an internal unmanaged CLR data structure that represents a running managed thread.  From
    there, they can pluck interesting information like the Thread::ThreadState bit field.  None
    of these data structures are part of our contract with managed applications and all
    of them are sure to change in future releases.  The
    only reason the ThreadState field is at a stable offset in our internal Thread struct
    today is that its frequency of access merits putting it near the top of the struct
    for good cache-line filling behavior.


  1. Reflection allows highly privileged code to access private
    members of arbitrary types.  I am aware
    of dozens of teams inside and outside of Microsoft which rely on this mechanism for
    shipping products.  Some of these uses
    are entirely justified, like the way Serialization accesses private state that the
    type author marked as [Serializable()].  Many
    other uses rather questionable, and a few are truly heinous.  Taken
    to the extreme, this technique converts every internal implementation detail into
    a publicly exposed API, with the obvious consequences for evolution and application


  1. Assembly loading and type resolution can happen on very
    different schedules, depending on how your application is running.  We
    ve seen applications that misbehave based on NGEN vs. JIT, domain-neutral vs. per-domain
    loading, and the degree to which the JIT inlines methods.  For
    example, one application created an AppDomain and started running code in it.  That
    code subsequently modified the private application directory and then attempted to
    load an assembly from that directory.  Of
    course, because of inlining the JIT had already attempted to load the assembly with
    the original application directory and had failed.  The
    correct solution here is to disallow any changes to an AppDomain s application directory
    after code starts executing inside that AppDomain.  This
    directory should only be modifiable during the initialization of the AppDomain.


  1. In prior blogs, I ve talked about unhandled exceptions
    and the CLR s default policy for dealing with them.  That
    policy is quite involved and hard to defend.  One
    aspect of it is that exceptions that escape the Finalizer thread or any ThreadPool
    threads are swallowed.  This keeps the
    process running, but it often leaves the application in an inconsistent state.  For
    example, locks may not have been released by the thread that took the exception, leading
    to subsequent hangs.  Now that the technology
    for reporting process crashes via Watson dumps is maturing, we really want to change
    our default policy for unhandled exceptions so that we Fail Fast with a process crash
    and a Watson upload.  However, any change
    to this policy will undoubtedly cause many existing applications to stop working.


  1. Despite the flexibility of CAS, most applications still
    run with Full Trust.  I truly believe
    that this will change over time.  For
    example, in Whidbey we will have ClickOnce permission elevation and in Longhorn we
    will deliver the Secure Execution Environment or SEE.  Both
    of these features were discussed at the PDC.  When
    we have substantial code executing in partial trust, we re going to see some unfortunate
    surprises.  For example, consider message
    pumping.  If a Single Threaded Apartment
    thread has some partial trust code on its stack when it blocks (e.g. Monitor.Enter
    on a contentious monitor), then we will pump messages on that thread while it is blocked.  If
    the dispatching of a message requires a stack walk to satisfy a security Full Demand,
    then the partially trusted code further back on the stack may trigger a security exception.  Another
    example is related to class constructors.  As
    you probably know, .cctor methods execute on the first thread that needs access to
    a class in a particular AppDomain.  If
    the .cctor must satisfy a security demand, the success of the .cctor now depends on
    the accident of what other code is active on the thread s stack.  Along
    the same lines, the .cctor method may fail if there is insufficient stack space left
    on the thread that happens to execute it.  These
    are all well understood problems and we have plans for fixing them.  But
    the fixes will necessarily change observable behavior for a class of applications.


I could fill a lot more pages with this sort of list.  And
our platform is still in its infancy.  Anyway,
one clear message from all this is that things will change and then applications will


But can we categorize these failures and make some sense
of it all?  For each failure, we need
to decide whether the platform or the application is at fault for each case.  And
then we need to identify some rules or mechanisms that can avoid these failures or
mitigate them.  I see four categories.



1:  The application explicitly screws

The easiest category to dispense with is the one where
a developer intentionally and explicitly takes advantage of a behavior that s/he knows
is guaranteed to change.  A perfect example
of this is #8 above.  Anyone who navigates
through private members to unmanaged internal data structures is setting himself up
for problems in future versions.  The
responsibility (or irresponsibility in this case) lies with the application.  In
my opinion, the platform should have no obligations.


But consider #5 above.  It
s clearly in this same category, and yet opinions on our larger team were quite divided
on whether we needed to fix the problem.   I
spoke to a number of people who definitely understood the incredible difficulty of
keeping this application running on new versions of the CLR and EnterpriseServices.  But
they consistently argued that the operating system has traditionally held itself to
this sort of compatibility bar, that this is one of the reasons for Windows ubiquity,
and that the managed platform must similarly step up.


Also, we have to be realistic here.  If
a customer issue like this involves one of our largest accounts, or has been escalated
through a very senior executive (a surprising number seem to reach Steve Ballmer),
then we re going to pull out all the stops on a fix or a temporary workaround.


In many cases, our side-by-side support is an adequate
and simple solution.  Customers can continue
to run problematic applications on their old bits, even though a new version of these
bits has also been installed.  For instance,
the config file for an application can specify an old version of the CLR.  Or
binding redirects could roll back a specific assembly.  But
this technique falls apart if the application is actually an add-in that is dynamically
loaded into a process like Internet Explorer or SQL Server.  It
s unrealistic to lock back the entire managed stack inside Internet Explorer (possibly
preventing newer applications that use generics or other Whidbey features from running
there), just so older questionable applications can keep running.


It s possible that we could provide lock back at finer-grained
scopes than the process scope in future versions of the CLR.  Indeed,
this is one of the areas being explored by our versioning team.


Anyway, if we were under sufficient pressure I could
imagine us building a one-time QFE (patch) for an important customer in this category,
to help them transition to a newer version and more maintainable programming techniques.  But
if you aren t a Fortune 100 company or Steve Ballmer s brother-in-law, I personally
hope we would be allowed to ignore any of your applications that are in this category.



2:  The platform explicitly screws the

I would put #6, #7 and #11 above in a separate category.  Here,
the platform team wants to make an intentional breaking change for some valid reason
like performance or reliability.  In fact,
#10 above is a very special case of this category.  In
#10, we would like to break compatibility in Whidbey so that we can provide a stronger
model that can avoid subsequent compatibility breakage.  It
s a paradoxical notion that we should break compatibility now so we can increase future
compatibility, but the approach really is sensible.


Anyway, if the platform makes a conscious decision to
break compatibility to achieve some greater goal, then the platform is responsible
for mitigation.  At a minimum, we should
provide a way for broken applications to obtain the old behavior, at least for some
transition period.  We have a few choices
in how to do this, and we re likely to pick one based on engineering feasibility,
the impact of a breakage, the likelihood of a breakage, and schedule pressure:


  • Rely on side-by-side and explicit administrator intervention.  In
    other words, the admin notices the application no longer works after a platform upgrade,
    so s/he authors a config file to lock the application back to the old platform bits.  This
    approach is problematic because it requires a human being to diagnose a problem and
    intervene.  Also, it has the problems
    I already mentioned with using side-by-side on processes like Internet Explorer or
    SQL Server.


  • For some changes, it shouldn t be necessary to lock
    back the entire platform stack.  Indeed,
    for many changes the platform could simultaneously support the old and new behaviors.  If
    we change our default policy for dealing with unhandled exceptions, we should definitely
    retain the old policy& at least for one release cycle.


  • If we expect a significant percentage of applications
    to break when we make a change, we should consider an opt-in policy for that change.  This
    eliminates the breakage and the human involvement in a fix.  In
    the case of String Interning, we require each assembly to opt-in to the new non-intern
    ed behavior.


  • In some cases, we ve toyed with the idea of having the
    opt-in be implicit with a recompile.  The
    logic here is that when an application is recompiled against new platform bits, it
    is presumably also tested against those new bits.  The
    developer, rather than the admin, will deal with any compatibility issues that arise.  We
    re well set up for this, since managed assemblies contain metadata giving us the version
    numbers of the CLR and the dependent assemblies they were compiled against.  Unfortunately,
    execution models like ASP.NET work against us here.  As
    you know, ASP.NET pages are recompiled automatically by the system based on dependency
    changes.  There is no developer available
    when this happens.




Before we look at the next two categories of AppCompat
failure, it s worth taking a very quick look at one of the techniques that the operating
system has traditionally used to deal with these issues.  Windows
has an AppCompat team which has built something called a shimming engine.


Consider what happened when the company tried to move
consumers from Win95/Win98/WinMe over to WinXP.  They
discovered a large number of programs which used the GetVersion or the preferred GetVersionEx
APIs in such a way that the programs refused to run on NT-based systems.


In fact, WinXP did such a good job of achieving compatibility
with Win9X systems that in many cases the only reason
the application wouldn t run was the version check that the program made at start
up.  The fix was to change GetVersion
or GetVersionEx to lie about the version number of the current operating system.  Of
course, this lie should only be told to programs that need the lie in order to work


I ve heard that this shim which lies about the operating
system version is the most commonly applied shim we have.  As
I understand it, at process launch the shimming engine tries to match the current
process against any entries in its database.  This
match could be based on the name, timestamp or size of the EXE, or of other files
found relative to that EXE like a BMP for the splash screen in a subdirectory.  The
entry in the database lists any shims that should be applied to the process, like
the one that lies about the version.  The
shimming engine typically bashes the IAT (import address table) of a DLL or EXE in
the process, so that its imports are bound to the shim rather than to the normal export
(e.g. Kernel32!GetVersionEx).  In addition,
the shimming engine has other tricks it perform less frequently, like wrapping COM
objects up with intercepting proxies.


It s easy to see how this infrastructure can allow applications
for Win95 to execute on WinXP.  However,
this approach has some drawbacks.  First,
it s rather labor-intensive.  Someone
has to debug the application, determine which shims will fix it, and then craft some
suitable matching criteria that will identify this application in the shimming database.  If
an appropriate shim doesn t already exist, it must be built.


In the best case, the application has some commercial
significance and Microsoft has done all the testing and shimming.  But
if the application is a line of business application that was created in a particular
company s IT department, Microsoft will never get its hands on it.  I
ve heard we re now allowing sophisticated IT departments to set up their own shimming
databases for their own applications but this only allows them to apply existing
shims to their applications.


And from my skewed point of view the worst part of
all this is that it really won t work for managed applications.  For
managed apps, binding is achieved through strong names, Fusion and the CLR loader.  Binding
is practically never achieved through DLL imports.


So it s instructive to look at some of the techniques
the operating system has traditionally used.  But
those techniques don t necessarily apply directly to our new problems.


Anyway, back to our categories&



3:  The application accidentally screws

4:  The platform accidentally screws the

Frankly, I m having trouble distinguishing these two
cases.  They are clearly distinct categories,
but it s a judgment call where to draw the line.  The
common theme here is that the platform has accidentally exposed some consistent behavior
which is not actually a guaranteed contract.  The
application implicitly acquires a dependency on this consistent behavior, and is broken
when the consistency is later lost.


In the nirvana of some future fully managed execution
environment, the platform and tools would never expose consistent behavior unless
it was part of a guarantee.  Let s look
at some examples and see how practical this is.


In example #1 above, reflection used to deliver members
in a stable order.  In Whidbey, that order
changes.  In hindsight, there s a simple
solution here.  V1 of the product could
have contained a testing mode that randomized the returned order.  This
would have exposed the developer to our actual guarantees, rather than to a stronger
accidental consistency.  Within the CLR,
we ve used this sort of technique to force us down code paths that otherwise wouldn
t be exercised.  For example, developers
on the CLR team all use NT-based (Unicode) systems and avoid Win9X (Ansi) systems.  So
our Win9X Ansi/Unicode wrappers wouldn t typically get tested by developers.  To
address this, our checked/debug CLR build originally considered the day of the week
and used Ansi code paths every other day.  But
imagine chasing a bug at
11:55 PM
.  When the bug magically disappears on
your next run at
1:03 AM
the next morning, you are far too frazzled to think clearly about the reason.  Today,
we tend to use low order bits in the size of an image like mscorwks.dll or the assembly
being tested, so our randomization is now more friendly to testing.


In example #2 above, you could imagine a similar perturbation
on our AutoLayout algorithms when executing a debug version of an application, or
when launched from inside a tool like Visual Studio.


For example #4, the CLR already has internal stress modes
that force different and aggressive GC schedules.  These
can guarantee compaction to increase the likelihood of detecting stale references.  They
can perform extensive checks of the integrity of the heap, to ensure that the write
barrier and other mechanisms are effective.  And
they can ensure that every instruction of JITted managed code that can synchronize
with the GC will synchronize with the GC.  I
suspect that these modes would do a partial job of eradicating assumptions about lifetimes
reported by the JIT.  However, we will
remain exposed to significantly different code generators (like Rotor s FJIT) or
execution on significantly different architectures (like CPUs with dramatically more


In contrast with the above difficulty, it s easy to
imagine adding a new GC stress mode that perturbs the finalization queues, to uncover
any hidden assumptions about finalization order.  This
would address example #3.



Customer Debug Probes, AppVerifier and other

It turns out that the CLR already has a partial mechanism
for enabling perturbation during testing and removing it on deployed applications.  This
mechanism is the Customer Debug Probes feature that we shipped in V1.1.  Adam
Nathan s excellent blog site has a series of articles on CDPs, which are collected
together at  The
original goal of CDPs was to counteract the black box nature of debugging certain
failures of managed applications, like corruptions of the GC heap or crashes due to
incorrect marshaling directives.  These
probes can automatically diagnose common application errors, like failing to keep
a marshaled delegate rooted so it won t be collected.  This
approach is so much easier than wading through dynamically generated code without
symbols, because we tell you exactly where your bugs are.  But
we re now realizing that we can also use CDPs to increase the future compatibility
of managed applications if we can perturb current behavior that is likely to change
in the future.

Unfortunately, example #6 from above reveals a major
drawback with the technique of perturbation.  When
we built the original implementation of Object.GetHashCode, we simply never considered
the difference between what we wanted to guarantee (hashing) and what we actually
delivered (OIDs).  In hindsight, it is
obvious.  But I m not convinced that
we aren t falling into similar traps in our new features.  We
might be a little smarter than we were five years ago, but only a little.


Example #10 worries me for similar reasons.  I
just don t think we were smart enough to predict that changing the binding configuration
of an AppDomain after starting to execute code in that AppDomain would be so fragile.  When
a developer delivers a feature, s/he needs to consider security, thread-safety, programming
model, key invariants of the code base like GC reporting, correctness, and so many
other aspects.  It would be amazing if
a developer consistently nailed each of these aspects for every new feature.  We
re kidding ourselves if we think that evolution and unintentional implicit contracts
will get adequate developer attention on every new feature.


Even if we had perfect foresight and sufficient resources
to add perturbation for all operations, we would still have a major problem.  We
can t necessarily rely on 3rd party developers to test their applications
with perturbation enabled.  Consider the
unmanaged AppVerifier experience.


The operating system has traditionally offered a dynamic
testing tool called AppVerifier which can diagnose many common unmanaged application
bugs.  For example, thanks to uploads
of Watson process dumps from the field, most unmanaged application crashes can now
be attributed to incorrect usage of dynamically allocated memory.  Yet
AppVerifier can use techniques like placing each allocation in its own page or leaving
pages unmapped after release, to deterministically catch overruns, double frees, and
reads or writes of freed memory.


In other words, there is hard evidence that if every
unmanaged application had just used the memory checking support of AppVerifier, then
two out of every three application crashes would be eliminated.  Clearly
this didn t happen.


Of course, AppVerifier can diagnose far more than just
memory problems.  And it s very easy
and convenient to use.


Since testing with AppVerifier is part of the Windows
Logo compliance program, you would expect that it s used fairly rigorously by ISVs.  And,
given its utility, you would expect that most IT organizations would use this tool
for their internal applications.  Unfortunately,
this isn t the case.  Many applications
submitted for the Windows Logo actually fail to launch under AppVerifier.  In
other words, they violate at least one of the rules before they finish initializing.


The Windows AppCompat team recognizes that proactive
tools like AppVerifier are so much better than reactive mitigation like shimming broken
applications out in the field.  That
s why they made the AppVerifier tool a major focus of their poorly attended Application
Compatibility talk that I sat in on at the PDC.  (Aha!  I
really was going somewhere with all this.)


There s got to be a reason why developers don t use
such a valuable tool.  In my opinion,
the reason is that AppVerifier is not integrated into Visual Studio.  If
the Debug Properties in VS allowed you to enable AppVerifier and CDP checks, we would
have much better uptake.  And if an integrated
project system and test system could monitor code coverage numbers, and suggest particular
test runs with particular probes enabled, we would be approaching nirvana.



Winding Down

Looking at development within Microsoft, one trend is
very clear:  Automated tools and processes
are a wonderful supplement for human developers.  Whether
we re talking about security, reliability, performance, application compatibility
or any other measure of software quality, we re now seeing that static and dynamic
analysis tools can give us guarantees that we will never obtain from human beings.  Bill
Gates touched on this during his PDC keynote, when he described our new tools for
statically verifying device driver correctness, for some definition of correctness.


This trend was very clear to me during the weeks I spent
on the DCOM / RPCSS security fire drill.  I
spent days looking at some clever marshaling code, eventually satisfying myself that
it worked perfectly.  Then someone else
wrote an automated attacker and discovered real flaws in just a few hours.  Other
architects and senior developers scrutinized different sections of the code.  Then
some researchers from MSR who are focused on automatic program validation ran their
latest tools over the same code and gave us step-by-step execution models that led
up to crashes.  Towards the end of the
fire drill, a virtuous cycle was established.  The
code reviewers noticed new categories of vulnerabilities.  Then
the researchers tried to evolve their tools to detect those vulnerabilities.  Aspects
of this process were very raw, so the tools sometimes produced a great deal of noise
in the form of false positives.  But it
s clear that we were getting real value from Day One and the future potential here
is enormous.


One question that always comes up, when we talk about
adding significant value to Visual Studio through additional tools, is whether Microsoft
should give away these tools.  It s a
contentious issue, and I find myself going backwards and forwards on it.  One
school of thought says that we should give away tools to promote the platform and
improve all the programs in the Windows ecology.  In
the case of tools that make our customers applications more secure or more resilient
to future changes in the platform, this is a compelling argument.  Another
school of thought says that Visual Studio is a profit center like any other part of
the company, and it needs the freedom to charge what the market will bear.


Given that my job is building a platform, you might expect
me to favor giving away Visual Studio.  But
I actually think the profit motive is a powerful mechanism for making our tools competitive.  If
Visual Studio doesn t have P&L responsibility, their offering will deteriorate
over time.  The best way to know whether
they ve done all they can to make the best tools possible, is to measure how much
their customers are willing to pay.  I
want Borland to compete with Microsoft on building the best tools at the best price,
and I want to be able to measure the results of that competition through revenue and
market penetration.


In all this, I have avoided really talking about the
issues of versioning.  Of course, versioning
and application compatibility are enormously intertwined.  Applications
break for many reasons, but the typical reason is that one component is now binding
to a new version of another component.  We
have a whole team of architects, gathered from around the company, who have been meeting
regularly for about a year to grapple with the problems of a complete managed versioning
story.  Unlike managed AppCompat, the
intellectual investment in managed versioning has been enormous.


Anyway, Application Compatibility remains a relatively
contentious subject over here.  There
s no question that it s a hugely important topic which will have a big impact on
the longevity of our platform.  But we
are still trying to develop techniques for achieving compatibility that will be more
successful than what Windows has done in the past, without limiting our ability to
innovate on what is still a very young execution engine and set of frameworks.  I
have deliberately avoided talking about what some of those techniques might be, in
part because our story remains incomplete.


Also, we won t realize how badly AppCompat will bite
us until we can see a lot of deployed applications that are breaking as we upgrade
the platform.  At that point, it s easier
to justify throwing more resources at the problem.  But
by then the genie is out of the bottle& the deployed applications will already
depend on brittle accidents of implementation, so recovery will be painfully breaking.  In
a world where we are always under intense resource and schedule pressure, the needs
of AppCompat must be balanced against performance, security, developer productivity,
reliability, innovation and all the other must haves .


You know, I really do want to talk about Hosting.  It
is a truly fascinating subject.  I m
much more comfortable talking about non-preemptive fiber scheduling than I am talking
about uninteresting topics like implicit contracts and compatibility trends.


But Hosting is going to have to wait at least a few more

Comments (31)

  1. Sriram says:

    Why does your RSS feed have only a truncated description for this post? Please let your entire posts stay in the RSS feed. I use a dial-up connection and read my aggregator’s new downloads offline….so this is cumbersome to say the least

  2. This is a fantastic post, and you are exactly right about App Compat being an ignored topic. I was one of the very few people at the app compat talk (and the only one in the front row with a badge that didn’t read Microsoft on it) as you were, and those that were there seemed disappointed in the lack of "sexiness" of this session. (The questions were along the lines of, "I was expecting to hear what you are doing for app compat in Longhorn…" "Well, we’re doing more of the same. Unlike most everything else here, you can use our bits today.") Anyway, I enjoyed your description of shimming, and I did not realize that it was done this way. Does anybody on the app compat team have a blog (akin to Raymond Chen’s brilliantly fun stories of the hows and whys of his knowledge domain)? Not only would it be fun, but if we’re not careful we just might learn someting. (Hey hey hey…) I remember the fun with Whistler, trying to get it to run Needs for Speed Porsche Unleashed, filing a bug against a DLL whose sole purpose in life was to grab as much memory as possible from the Win95 heap, which would eventually fail and the process would keep going. Whistler would never fail, and the machine would just grind to a halt, and the only solution was to drop in a Win95 heap emulator.

  3. Chris Brumme says:

    Sorry Sriram. I’ve corrected my mistake.

  4. Junfeng says:

    Chris, Raymond Chen worked in app compat team for many years. He is your man;)

  5. Eric Wilson says:

    Once again Chris, you do not disappoint us. The only thing that annoys me is that we do not get MORE posts:) I swear that I have read your post on Application Shutdown at least 4 times start to finish. I’m facinated by all the topics you have up here. Keep the great posts coming!

  6. Ian Ringrose says:

    Start quote –>
    There s got to be a reason why developers don t use such a valuable tool. In my opinion, the reason is that AppVerifier is not integrated into Visual Studio. If the Debug Properties in VS allowed you to enable AppVerifier and CDP checks, we would have much better uptake. And if an integrated project system and test system could monitor code coverage numbers, and suggest particular test runs with particular probes enabled, we would be approaching nirvana.
    <– End quote.

    a) First off, every place I have worked at, I get asked by managers’ match more often about shipping date then quality of the software.

    b) I have NEVER come across a customer running AppVerifier as part of the evaluation process before buying an application. Does EVERYONE at Microsoft do so before buying in software?

    c) The fact that a application that a ISV ships 2 years ago does not work on the next version of Windows is a good thing for the ISV:-) It makes it easer to sell support contracts, and to get customers to pay for upgrades. E.g. Exchange 2000 not working on NT 2003.

    So what can be done?
    ) Make it very hard, not to run AppVerifier and other such tools when debugging an application in msdev. How about a pop up the points up that the developer, not his boss may land up being sued if he turns of AppVerifier?

    b) When a user installs a new application, Windows should ask if the user wish to test it with AppVerifier for the first week of usage, there should be NO way that the application vender (OR IT department) can turn this off!

    c) All systems that ship with version 1.1 (and 1.2) of the frameworks should also contains version 1.0 (and 1.1). By default an application should ONLY bind with the version it was built with. I do not see way each web page could not use a different version of the Framework, likewise with each SQL server database.

    Email: ringi at bigfoot dot com

  7. Jim Argeropoulos says:

    Chris I admit having not used AppVerifier in my unmanaged days. I did make heavy use of BoundsChecker.

    My question is this, What tools should I be using today for my managed applications. fxCop certainly falls on this list , but what else have I missed?


  8. Any plans for tightening the security model by switching to execution history-based access control instead of crawling the stack?

  9. Alek says:

    Dont you think that MS VS competing with any other tool vendor is unfair competition? I am disappointed to see this presented as a good thing …

  10. Keith Hill says:

    Comment to Alek: don’t you think MS giving away VS would be even more unfair? Or do you think MS shouldn’t provide tools for their platform? I do agree that folks like Borland are at a disadvantage for a number of reasons but the most I think MS can do about that is provide open/free access to the relevant APIs. WRT app compatibilty I hadn’t realized the issue with ASP.NET. I was always in favor of side-by-side is a great solution for binary compatibility. However, when it comes to source compatibility, I don’t mind breaking changes if they improve the platform and there is some way to do what I was doing before by tweaking the source code. I just really hate to see a young platform already start to take on the Foobar2 and FoobarEx type names. 🙁

  11. Chris Brumme says:


    I don’t think we would want to use legal or other coercion to force either developers or users to run with AppVerifier or some eventual managed equivalent. And although it’s reasonable for users to test potential products with AppVerifier before making a purchase, these tools are really intended for developers.

    I think our best approach is to make the tools so easy to use, and so valuable to use, that developers naturally do so. Compilers give you a lot of useful information about what might go wrong when you execute your application. Imagine if, on the first F5 in Visual Studio, you got a report of all the things that are a little flakey and questionable, with suggestions of exactly where and how to fix them.

    If the verification is completely integrated and provides compelling value, no coercion will be necessary.

  12. Chris Brumme says:


    If you are using FxCop today, that is certainly the single best thing you can do. You might also have a look at the ClrSpy utility that Adam Nathan built and describes in his blog. This is a convenient way to use Customer Debug Probes. See

    At some point, I think Microsoft needs to provide more FxCop rules, more Customer Debug Probes, new tools for detecting versioning changes, more diagnostic traces out of the managed class libraries, etc. But there’s also an opportunity for 3rd parties to add value here, too.

  13. Chris Brumme says:


    You ask about security using execution-based history rather than the heuristic of stack crawls. We occasionally run into circumstances where stack crawls don’t work well. I’ve talked about some of these cases in prior blogs, like async points and delegates. I think you’ll see us continue to address these issues in the context of stack crawls (i.e. by considering additional logical pieces of stack, as we have done since V1).

    For other cases, like privacy of data, the execution of code is a poor indicator of how information has been leaked across trust boundaries. We’ve had some conversations about how a different approach might be necessary for managed enforcement of privacy. But none of those ideas has led anywhere practical yet. I think if you scan the public research papers on this topic, you’ll come to that same conclusion.

    There are still plenty of interesting problems waiting to be solved.

  14. Steve Bolton says:

    Another excellent and thoughtful article. Blogs like yours have improved my opinion of MS enormously. I was inspired to download AppVerifier immediately and run it on Excel and Word 2003! (No this is not a cheap attack on MS here, rather I was interested to see what would turn up and they are big applications that I believe contain a lot of legacy code).
    I believe these tools should be included in Visual Studio with a lot more promotion about their use as you describe. Thanks.

  15. Mark Morrell says:

    For Chris, Alek, and Keith –

    Economically speaking, there would normally be an advantage (to Microsoft) of giving away Visual Studio and assocated tools to make developed software better. VS is a complementary product to windows, and use of VS would theoretically increase demand for Windows because it increases the quality of software developed for Windows. This makes economic sense because it results in higher profits by selling more copies of Windows.

    However, because Microsoft is considered a monopoly it doesn’t make sense to give VS away because you probably won’t increase demand by an appreciable amount anyway. Also, it would be considered unfair use of monopoly power because it is anti-competitive so the government would probably shut the effort down.

    Anyway, your best bet would be to bundle the error checking and compatibility technology you are talking about into VS without increasing the price of VS. These are excellent features that would help justify a new upgrade that the public would pay for, increasing revenue without running afoul of the DoJ.

    It’s really funny to me that being declared a monopoly, it would be considerd unethical for MS to give stuff away, but it’s fine to sell it. Like an inverse charity.

  16. Albert says:

    I think the keynote from MSR on Wednesday was the best keynote on the PDC. I watched the reactions from the crowd and this keynote received the most applauses. I’m sure that it could’ve been better but still it wasn’t by any means substandard.

  17. eAndy says:

    I love the information posted in your blog. It always seems too long between posts. (can’t get enough)

    None the less, I agree that you can’t give away VS for a number of reasons
    – current anti-competitive ‘stuff’
    – loss of long term value proposition (hard to keep investing $ in a product you give away free, you start to see less innovation)

    As to the tools that make for better programs such as
    – CDP
    – AppVerifier
    – (unnamed tools from MSR)

    these should continue to be free. You CANNOT underestimate the cost to MS image as a platform if you allow bad code to be written and deployed.

    You must make these tools accessible and free.

    As for VS.NET keep charging but increase the value of VS by integrating those free tools (CDP, AppVerififer, unnamed MSR tools) into the VS product. That’s the value add we’re paying for.

    Borland and others can continue to compete and leverage the tools as they see fit.

    You get competition and good code (which helps the platform )

    If you start charging for the tools that make better code, people don’t use them and you start to hear stupid stuff like "MS doesn’t scale" "MS isn’t reliable, it always crashes", "MS isn’t secure". Things that are attributed to bad coding practices rather than the platform

  18. ?????Blog ?????????

    ?????? ??3?????? Apartments and Pumping in the CLR ???

    ?????????????????????(^_^;) ????????????????????

  19. See Win App says:

    This post is actually a re-post of a post I did a little under year ago during PDC ’05 after attending…