internal = public?


There was an interesting thread this week inside the firewall about compatibility concerns the .NET Framework (and by extension WinFX) has in light of the fact that you can you use reflection to find and even invoke private members.  Here is the exact question:


What is our BREAKING CHANGES policy towards applications that call internal methods in our assemblies through Reflection? Are we obligated to be backwards compatible with such apps between versions, meaning no changes to internal methods (spec or behavior wise) in our assemblies?


 


Writing an application that depends on internal implementation details, even if they happen to be exposes via reflection, is not a very wise thing.  Even the book on the subject Hijacking .Net makes it clear that this technique should be very carefully used.


That said, other folks on the thread pointed out that judging from the QFEs and other customer reports we get an increasing number of folks are using private reflection in really important applications.  Does this mean even implementation details of the framework have to remain unchanged across versions?  That would seriously impair our ability to improve the platform over time and I don’t think this is our customer’s interests.  For those of you using private reflection to access implementation details of the framework do you expect those applications to keep working on newer versions of the .NET Framework or would you rather see us keep even internal details unchanged over time?  


 


On the CLR team, we’ve speculated (just speculation, no firm plans here at all) about whether we can make a breaking change to restrict any use of Reflection MemberAccess to some limited supportable scenarios.  For instance, we might prevent serializers from using private reflection technique except on members that are explicitly earmarked for serialization.    We would love to get any information you have on how your customers (or your own products) are actually using private reflection.  If anyone knows of such cases, please comment of drop me a line with any details you have. 

Comments (58)

  1. Drew Marsh says:

    Microsoft should not even waste their time worrying about this. It should be painfully obvious to anyone that resorts to reflection hacks that they are unsupported. Please focus on more important things and let the hackers who resort to such foolish techniques suffer when their code breaks.

    Just my 2¢.

  2. Louis Parks says:

    Why is it even possible to do this? What’s the point of scope access keywords, if they aren’t enforced by the runtime?

  3. Shane King says:

    Focus on exposing more frunctionality so that people no longer need to use reflection to access private members, rather than trying to stop code from breaking.

    It should be obvious to any developer that if you grab private members, you could break if those members disappear.

  4. Don Alvarez says:

    I can’t disagree more with the earlier comment that the ability to reflect private members is non-issue.

    The decision to allow public access to private members via reflection was an INCREDIBLY bad choice because it affects everyone who writes code for this environment. We _all_ have to worry about whether some idiot (uninformed or malicious) will break _our_ code via reflected access to the a private method. Having to worry about missuse of private methods every time you design a class is a phenomenal waste of developer time (either when you design the class or when you have to fix the pieces later).

    I wholeheartedly support the proposal to limit reflection of private members to an explicitly marked subset.

  5. Darrell says:

    Do not limit the future flexibility of the framework because a certain few people use reflection to access private-scope methods/variables. They can use side-by-side versioning features to keep their app working.

  6. Eric says:

    Don mentioned above that it was an INCREDIBLY bad choice. I don’t disagree; it is too easy and when I first learned about it I wondered why they even bothered allowing it. I would support this functionality being removed.

    That said, you can always have your code hijacked. Someone hosting the CLR or hooking it or hacking it can always get at your implementation. They can override security and change the metadata of your assembly. Just as they can have unsafe code and call native code.

    It appears to me that the real issue (and this is just opinion based on my experience), is that Code Access Security hasn’t panned out as well as it could. There’s no incentive for an app writer to write an application that has less than full trust, because users are used to native apps that need full trust anyway.

    Hopefully Longhorn really starts addressing this, as managed code proliferates and things like ClickOnce increase the demand for partial-trust apps. We’ll see though; customers don’t like complicated security stories, and are eager to override secure defaults if it lets them run an app they want to see.

  7. milbertus says:

    I would also agree with you shouldn’t be allowed to access private methods by any means, including reflection.

    As for introducing a breaking change by fixing this, I would say go for it. I would hope that anyone using reflection to get at private methods would have enough sense to realize that this may or may not work in the future. Besides, if they absolutely positively must use some random private method in the BCL, then let them stay with v1.1 of the Framework.

  8. Brad Abrams says:

    One point that keeps coming up is that if folks use private reflection they are OK if they just stick with the same version of the framework. Well, maybe, but consider Service Packs and other "hot" fixes we do… any of those could break an app using private reflection as well and those don’t go side-by-side generally.

  9. I have only encountered the need for private reflection when using the .NET Framework or VS.NET and not any other scenario. I believe that if you guys did a questionaire about which functionality people would like exposed by the Framework and IDE, it would remove the need for private reflection.

    I do not believe private reflection is needed or right when you program and that it is quite hurtful if you use it, but the components we write on my team that do use private reflection (minimal use, of course) are marked as compatible with the single version of the Framework or VS.NET that we know would surely work with. Upgrading would require us to intervene.

  10. Aaron Randolph says:

    If it breaks their stuff, tough luck. Take a lesson from all the posts Raymond has made about how Windows had to make special cases for applications that did undocumented things, and read the replies to those posts. It irks me that hacks had to be put in to keep those apps happy. I don’t want to see .NET go the same way.

    And I agree that internal methods should not be allowed to be reflected on. Isn’t the whole idea of internal or private so that they can’t be accessed except by the object containing them? So that other people/objects can’t call them and possibly screw something up internally?

  11. We used reflection to tweak your private and internal bits in at least ten places in a large COTS app to work around bugs in the .NET Framework code.

    I expect these fixes will break when the framework changes. No biggie, provided that you fix the bugs that caused me to do this.

    Just keep in mind that not everybody that’s doing this is evil. We just didn’t have much choice.

    Dejan

  12. Sami Vaaraniemi says:

    I think internal, protected and private methods *should* be allowed to be reflected on. After all, reflection is not about encapsulated access to an object, it is about looking at the internal details of an object. If you want to stick to safe, encapsulated OO programming practices, use the public interface only.

    At the same time, it should be made clear that any code that uses private implementation through reflection can and most likely will break without further notice.

    Uninformed and malicious users have always been around. Even if reflection may make their job of wreaking havoc marginally easier, I don’t think it changes it that much. They were able to wreak havoc with our code even in the unmanaged world.

  13. RichB says:

    I’ve said it before, but it’s worth saying in the context of this discussion.

    I believe that every single virtual method should be able to be re-implemented easily by a derived class.

    There have been several times when I have needed to add a single extra line of code in the middle of a .Net Framework method. I can derive from the class and override the method, but re-implementing the method is impossible (I’m thinking specifically about deriving from the ASP.Net page class and reimplementing some of the methods).

    In this case, the only way to achieve what I need is to use reflection. I think when people use reflection, they realize they are doing something which may break in future – however they also have no other choice.

  14. moo says:

    SO how do you propse we write profiles and debugging tools if we cannot access internal private stuff without reflection?

    It fine as it is, just theyre on theyre own if they do so.

    Bloody whiners crying over something thats ok as it is.

  15. W Poust says:

    I would definitely support not being able to reflect over private members unless they are marked with some kind of reflectable attribute.

  16. *Please* don’t try to maintain backwards compatibility for code that reflects against implementation details.

    I would be happier if you deliberately changed the names of all private members with each new version of the .NET Framework – that way you would establish a good precedent: everyone would then *know* their code is definitely going to break when the next version comes out. (It’d be reasonable to maintain the names across service packs on a particular major.minor release though – that way at least we can rely on side-by-side framework installation in the scenarios where we are forced to use such crufty code…)

    Of course the obvious way to change all the names would be to run the .NET Framework through an obfuscator. I’d prefer it if you didn’t do that though – the ability to grok the FCL with ILDASM has certainly made it *much* easier for me to learn how to use .NET. I’d hate that to go away in future versions.

    Also, I disagree with those who argue that we shouldn’t be able to reflect against these things in the first place. There are legitimate uses for this, as some have already pointed out in this discussion; I won’t repeat them, I just wanted to cast my -1 here.

  17. moo says:

    For those wanting no privates in reflect, what do you intend for use for an application profiler or debugger because you wont have any if you do that.

    Clueless dumbasses.

  18. Panos Theofanopoulos says:

    I dont agree

    I have a few cases of private access for the same reasons with Dejan and RichB (fix FCL bugs). Those may be fixed, in a future version but i am sure that others will surface. Since MS insist on not providing the FCL source code (irrational imho) must provide alternatives.

    Ian : I hope that Mono will be mature enough, the day that MS will release an obfuscated FCL 😉

  19. Koji Ishii says:

    I agree that MS is free to change private/internal implementation and is not responsible for apps who rely on them.

    But on the other hand, I don’t want reflection to private members prohibited. I have classes I want their constructors private. Within the assembly, I have a class that creates one of those classes. The class has a list of Type, and invokes constructors via reflection. This is a valid use of private reflection, isn’t this.

  20. moo says:

    Again, what do you expect to use to get access to the private members in a debugger? How do you profile these if theyre not available via reflection? You have yet to answer that. Well I say do it then, remove it and you can code blind. Lets see how far you get.

  21. James Bellinger says:

    I suggest allowing access to these methods if and only if the methods are contained in a debug-mode assembly (and include attributes — maybe one called ReflectionAccessAttribute… that way one could prevent even the enumeration of method names if they wanted).

    Past this, intentionally break applications which call internal methods. In the short term it will be unpleasant (but you have versioning, as has been mentioned, so it isn’t that big of an issue) but if you don’t do this you’re in for a world of hurt, as developers will then get the idea that it’s a safe thing to do. If code which calls internal methods breaks consistently, it will quickly become obvious that calling internal methods is a big no-no. Some have mentioned that calling internal methods can help work around bugs. That may be so, but working around bugs in this manner, in the long term, hurts the Framework as a whole. It will become dirty and unchangeable. Yes, it helps one developer, but it does not help the entire developer community.

  22. Martin says:

    My 0.02$:

    Please don’t limit Reflection of the Framework in any way – and don’t obfuscate it either. There are too many good reasons to keep it the way it is, e.g.:

    – Being able to use Roeders Reflector or similar tools to look at the decompiled source, to understand how things really work.

    – Being able to write debugging, profiling, serializing libs and tools – and lots of other, not yet invented programming tools.

    – Being able to work around bugs (erm, "issues" .. ;-P).

    Don’t worry too much about breaking someones code, if they access internal stuff. Internal means to me "don’t touch", so you can change it anytime – well, almost ..

    If i call or access something internal, i am either an idiot and deserve to learn thru suffering, or i am extremely desparate, because its the only possible way to solve some problem. In the latter case, i should notify the Framework authors about my problem, tell them about my terrible hack, and suggest a better solution for the next version.

    To avoid crashing at the poor innocent customers desktop, i should guard hacks like that by a Framework version check, e.g. something like this:

    if (version == 1.1.0.x) {

    call_internal_foobar (); // $HACK$ temporary workaround for bug xy in class System.FooBar …

    } else {

    throw new OopsException ("Unfortunately, this App requires .NET 1.1.0.x");

    }

    That means, if the Framework version changes from 1.1.0.x to 1.1.0.y, you should change internal stuff only if really you have to (e.g. to fix a bug), but not just because it would be cool or nifty.

    But if the version changes from 1.x to 1.y or even more, you’re completely free to turn everything (internal) upside down.

  23. bushidocoder says:

    IMHO, breaking the ability to call internal/protected/private members via reflection would be a horrible move because it would break most debuggers and unit testing packages. The earlier idea to make those calls only available against debug assemblies is interesting, but it might make profiling a bit more difficult in some circumstances.

    The worst case scenario in my mind would be creating an environment where developers started using the wrong level of data hiding just so that their development tools would work – internal/protected/private already have meaning, so don’t add additional meaning to them.

    Perhaps the answer would be to create a new system-level attribute which you could apply to methods (or classes on the whole) that would explicitly deny the ability to reference hidden members via reflection in release-build assemblies. This allows the developers who want to lockdown their code the power to do so, while not infringing on the power the framework gives those of us who 9 times out of 10 don’t give a damn.

    -bc

  24. Sam Meldrum says:

    Don’t break the ability to call private methods! But also, do not worry about breaking apps that do this. Anyone who has done this has done it in the full knowledge that what they are doing is not what was intended.

    Concentrate most on removing the reasons making us want to do this. The only time I have ever done this in production code is to get round a bug (/ problem) in the framework classes.

    E.g.

    the documentation for TreeNode.Collapse says:

    "The Collapse method only collapses the current TreeNode; child tree nodes are not collapsed."

    This is not what happens in practice (at least not in framework 1.0), in practice child tree nodes are also collapsed. Which is bizarre, since this implementation is more code and harder work than the simpler as per documentation case. Take a look at the IL – why does it do that?

    And if you want to just collapse the node and not the children, without manually recording the state of all child nodes pre-collpase and restoring post-collapse, the simplest implementation is something like this:

    MethodInfo mi = typeof(TreeNode).GetMethod("DoCollapse",

    BindingFlags.Instance | BindingFlags.NonPublic,

    null, new Type[]{typeof(TreeView)}, null);

    if (mi != null) {

    mi.Invoke(node, new object[]{this});

    }

    Now. If that sort of bug wasn’t there, I’d have no reason to go calling private methods. I’m prefectly prepared for my code to break in future versions of the framework, there’s an easy fix, and my code documentation tells me exactly what I’ve done and why. But if it breaks because you’ve taken away my ability to call private methods, rather than because you’ve fixed the original bug, I will be irritated.

    Regards,

    Sam

  25. Jim Arnold says:

    +1 for not disabling private reflection. Instead, fix the bugs and/or expose the functionality people are trying to use as public, supported code. Reflection is not the problem, it’s the sealed/non-virtual/private nature of so much of the framework classes.

    Jim

  26. moo says:

    If you disable that you ruin my buisness products, I will no longer support my developer tools on .NET and C#.

    I will move my products away from .NET.

  27. moo says:

    We make developer tools and profilers and debugging tools, I guess you dont want those if you do this.

    Its ok, we can get money from non .NET houses.

  28. moo says:

    We make developer tools and profilers and debugging tools, I guess you dont want those if you do this.

    Its ok, we can get money from non .NET houses.

  29. nfactorial says:

    I don’t see why support for debugging/profiling tools couldn’t be handled in a similar way to how these tools worked for unmanaged C++. By emitting (or exposing) the relevant information with an ‘export debug information’ option. Though as I don’t write such tools, I don’t know if that solves all your problems (and it doesn’t sound like a minor change either).

    I currently use reflection for enumerating and creating classes that implement specific interfaces (for plug-in support). In my own plugins I generally flag them as internal (so they are not easily exposed simply by linking to the assembly). Though I personally wouldn’t mind if the ability to reflect on privateinternal items was restrictedchangedmoved, some way of exposing items would be appreciated (I’d guess an ‘EnableReflectionAttribute’ of some-sort) for times when you *do* actually want them to be reflected dispite their access flags.

    Indeed, the ‘export debug information’ option I mentioned earlier could simply mean all items are implicitly flagged with the attribute. When a release build is made such information is removed.

    n!

  30. moo says:

    Kill reflection because some asshats cant handle the language and tools will kill the language. I for one will jump ship.

  31. moo says:

    Reflection is one of the many good aspects of having C# and the CLR.

  32. nfactorial says:

    I don’t think anyone mentioned ‘killing’ reflection?

    n!

  33. moo says:

    Well, when they start about saying "Only let it on debug builds" there goes profiling and debugging release builds.

    SO yeah that would kill it in some scenarios. Then they say use attributes to enable it, well thats not the same code one ships, that rules out that scenario, its dead there.

    So, yeah in a way it would.

  34. nfactorial says:

    That was me who mentioned debug builds? And I didn’t mean ‘debug builds’ only, maybe I should rephrase myself. I meant ‘debug info’ which is entirely different (you can have debug info in release builds) and it was only a suggestion. It could be implemented in other ways too, having a special assembly flag that make all items reflectable for instance (though it would have to be more secure than a simple flag, to prevent anyone simply setting it).

    I fail to see how attributes affect the code that gets shipped, it doesn’t affect the code-paths taken during runtime (except for the attribute constructor called by the ‘host’ application when the attributes are accessed). As I said, even if that’s still not good enough, there are other ways.

    I thought the original point was that reflection enables people to do bad things, what do people think of limiting reflection. Even to the scope of having coders explicitly tag privateinternal items as ‘reflectable’ (which, I guess, implies the use of an attribute). The majority of the time I add a privateinternal item I *really* do want it privateinternal (ie. I would prefer it couldn’t be reflected). Having the option would certainly make me happy and give more control over the assembly to the developer.

    n!

  35. moo says:

    Any changes to code mean its a different product mathmatically.

    It may be similar but infact its not what you ship, we all know the bugs that arise from a release build that dont show up in a debug build on the unmanaged world for example due to the fact that memory layout is different.

    Simple fact is, its not broken, it works as it should and the only people that I see complaining are those that cannot understand the tools and want Develop By Numbers by Big Bird.

  36. nfactorial says:

    Yes, but adding an attribute does not change the code. It does add new attributes of course, but the executing of the application is not affected.

    If you’re simply referring to the fact that the attribute is taking up memory and thus has changed the memory layout of the application, it should be noted the JIT process amongst other things makes the memory layout extremely volatile anyway. Saying all that, there are other ways of doing that doesn’t even require attributes applied to everything.

    I’m not sure if your last statement was for or against something 🙂

    n!

  37. moo says:

    If you dumb down the language enough you will only have dumb people using it.

    Do you really want that 😀

  38. moo says:

    There is already languages for people like that, its called BBC Basic

  39. moo says:

    Actually you will have worse, you will see a fragmentation of C# on different runtimes, yours vs everybody elses. We will either go our own C# and sod u lot or we will just not implement those features.

    Watch Mono et all part from the ECMA or start submitting theyre own drafts.

  40. nfactorial says:

    I’m not sure where anyone has mentioned ‘dumbing’ down the language, it’s simply offering more control to the developer (which, if you think about it, makes it more complex).

    n!

  41. moo says:

    As for the JIT affecting memory layout , its deterministic like youre program. If you follow the same code path etc you will have the same memory layout

  42. Dewayne Christensen says:

    Ian++.

  43. nfactorial says:

    The JIT doesn’t guarantee deterministic memory layout, it can put your code anywhere it likes.

    And this is all veering steeply away from the topic, there are other ways to provide such debug information without adding attributes everywhere.

    n!

  44. moo says:

    Yes but if everything was done EXACTLY the same as the last run it would end up the same. System was the same, state the same the hoopla exactly the same yes it would. If not then its a RANDOM JIT and no Logic whatsoever.

  45. Well, just to add to the cacophony..

    I think relfection of private/internal has its uses, but I don’t think MS should think twice about breaking changes for those. Don’t hold all your customers back for the few that are using non-public interfaces extensively.

  46. old man says:

    Breaking changes in this case actually sounds like a good idea to me. The multiple benefits including keeping the framework flexible, motivating better bug reporting, motivating better programming habits, etc.

    In a similar vein, I prefer keeping reflection available to those who wish to use it. Intentionally limiting the power of any tool seems to me the surest way to lose its developer base.

  47. moo says:

    As soon as I see C# becomming dummed down, I walk or I split the compiler using other sources (mono or whatever).

  48. Ian Ringrose says:

    I do not mind you breaking code that uses reflection when you do a major new release, .e.g. version 1.2. You should however not break such code when you do a service pack.

    The questions is, was 1.1 a service pack, or a major new release.

    I do not even mind you change some of the APIs for major release, a day or two to porting to get my code to work with version 1.2 is not a problem. Provided that the changes I need to made also work with version 1.1 (using #if is OK at times).

  49. moo says:

    I guess if I break open my car powertrain control computer and hack it they would support me by this methadology also.

    Im on my own when I do that.

    The latest IDE feature is a way to HIDE things from the debugger apparently, WTF.

    Now I can HIDE my bugs! YAY!

    Tardlicious!!! What genius thought up this idea.

  50. moo says:

    This crap belongs at the IDE level not the language level.

    The latest brainwaves coming out of Microsoft the past few years is just getting to the point of stupidity and it shows on the spasticated UIs on XP, and Longhorn.

  51. I don’t see how limiting reflection is going to help. If some "clever" person is trying to get a private member and Reflection stops them, they’ll move into writing unsafe code and poking around in memory to find what they’re looking for. Read Raymond Chen’s blog on the wicked undocumented things some people have done that’s introduced all sorts of nasty things into Windows.

    To any customer who starts whining because they knowingly did reflected on an implementation detail, tell ’em to go * themselves.

  52. moo says:

    Great so now profilers and debuggers are now "hacks" and "kludges" and "workarounds".

    I guess you want to pay the upgrade fee every time they change something.

    Good for you Mr.Mensa

  53. Andrew Fung says:

    1) Please don’t break reflection on privates. When I write a class, I may write it for the problem domain, and hide implementation details. However, if I write something that similarly solves a business problem (e.g. persistence) but needs access to implementation details (e.g. fields), without private reflection, I have to modify the API in a way that does nothing for directly solving the business problem the class was designed for.

    2) I’m knowingly walking around a visibility decision that the API designer made. That should be enough warning so that I should be aware and amenable to internal changes. So go ahead, change as you please. If I encounter a sign saying "construction ahead, things may change", I have no right to making presumptions on how things ahead might look.

    3) In terms of security and mucking around where you don’t want people to go, I’m not sure how the CLR does it, but as I recall, Java gives full security permissions when an application is executed locally, and limited permissions for code being executed in a different way (e.g. Activation). I would assume the basis is that if the caller has access to the binary file, any visibility protection isn’t true protection, merely an obstacle. There’s really nothing you can do against someone who can change what gets run (contrast to code that was downloaded and is being allowed to run locally). Obfuscation would probably be a better obstacle.

    4) If you *have* to go the way of trying to limit access, I would rather turn it around and have the default as it is, and specify some attribute to indicate to the CLR not to reflect on a member. Then people who have a concrete reason for doing so can do so, while people who don’t have a good reason don’t break everyone else by lack of foresight.

    5) And my last point (sorry, I’m wordy), the CLR could set permissions on startup based on the user’s permission in the OS. If the user doesn’t have write privileges on the .EXE being run, then visibility control will actually be hard to get around. This shifts the burden into deployment, but at least the possibility is still there in a way that still permits common, justifiable uses nowadays to continue.

    My $0.05c

  54. Dmitriy Zaslavskiy says:

    Catching up with my blog reading.

    In my project I used reflection against private/internal members when MS code had memory leaks and were way to slow to provide QFE(s). I surrounded my reflection code with all kinds of checks to backout gracefully if something goes wrong, even though it’s not full proof.

    Despite all this I would be more then willing to fix my code not to use reflection if MS provides QFEs, SPs to address my issues.

    Bottom line: improve platform/ignore violators

  55. Math says:

    Beeing able to access private members of a class is plain out silly.

    I dont understand how any sane person could design the framework that way.