I suppose it's a bit like that film The Matrix - you realize that you live in an ethereal and closed world only when you actually get to step outside of it. Or, like some people who have never been to another country, your view of the rest of the world is shaped just by what you see on TV. I guess I've been like that with open source stuff, and particularly Java; looking out incredulously from my little village of Microsoft technologies and products at the wide world beyond.
I started my computing days with what we euphemistically called a "home computer" (basically a games console with a keyboard), and progressed via a series of Amstrad computers to real PCs. At the time I was doing statistical and reporting work for a large manufacturing company, and had played with several MSDOS-based databases until I finally found nirvana with Windows 3.1 and Microsoft Access 1.0.
OK, so I'd learned a lot of computing theory in the meantime (such as a mixture of languages and programming theory), and I'd even written and sold technical, commercial, and business software. But most of it, especially after drifting deeper into the Windows way, was aimed at Microsoft operating systems and integration with Microsoft products. Gradually I'd been drawn in and captured by the Redmond magic.
It's only since I began work on our current HDInsight project that I've had to navigate deeper into the dark and scary jungles of open source; slashing away at the undergrowth of bewildering terminology with my virtual machete (Bing); wading knee-deep through murky and meandering streams of sometimes conflicting advice and guidance; and peering in amazement at the vast array of previously undiscovered wonders of nature such as ants, pigs, hives, zebra, and even a strange yellow elephant.
Sitting inside my nice comfy and well-defined Microsoft technology world, it's un-nerving to realize that until recently I never knew that all of this even existed. OK, so I've had dealings with Java, though mainly only under duress, and I've read about and even learnt a bit of other languages and frameworks such as Python and Ruby. In fact my first serious attempts at creating Windows DLLs were with Pascal (mainly because I could never get my head round C++). So Java code itself isn't really an issue.
No, where it all gets complicated is that almost all of the docs I read about Hadoop and the associated technologies are written by experts for experts. It seems like you need to know all about a whole range of topics and technologies before you can start learning about them. It's a bit like letting someone watch a medical drama series on TV and then giving them a scalpel, an operating theatre, and some (currently) live patients to practice on.
For example, I read endless articles about testing and debugging. It seems I should start by mocking out my objects (makes sense) and use a test runner to execute them within a single node local installation of Hadoop and with the Java virtual machine in advanced debug mode (I think). So I read about PowerMock, but it says I should use it with Mockito, but that says it's an extension of EasyMock. And I probably want to do it all from within Eclipse.
And to set up a local cluster I need to install Hadoop directly, although I have the HDInsight single node development environment already installed on my laptop. Can I use that instead? And if I want to run the Java VM in debug mode I probably want to use another program called ant to configure it. I'm sure I'll work all this out in time, though at the moment I'm wondering if it's going to be easier just to phone a friend (or ask the audience).
What's clear, however, is just how adaptable, configurable, and interoperable all this stuff is. I suppose I'm used to a strict product hierarchy and road map, nice dialog boxes and configuration web pages, and reams of beginner documentation for the Microsoft technologies that make up the confines of my own little world. Now I've strayed outside of the Microsoft Matrix, and discovered a whole wide world out there, I can't find a map. I'll probably spend the rest of my writing days blundering through the undergrowth, peering hopefully at every new edifice of civilization I can find, until - hopefully - it all start to fall into place. Or until some kind soul (aka Program Manager) taps me on the shoulder and mutters the equivalent of "Dr. Livingstone I presume."
Yes, the adventure is fun at the moment, but I'm quite looking forward to going home again...