I live in Berkeley. Most of the houses here pre-date me. Almost all are built with wooden frames, which means almost all were built using nails. Imagine, if you haven't already, that the Naitional Nail Authority has issued Nail 2.0. Immediately, however, a bug in the Nail 2.0 implementation causes the underlying Nail 1.0 implementation to AB-END and fail to build. So immediately, of course, all Nail 1.0 and later spec nails cease to exist. So 95% of the houses in Berkely immediately fall over, and the town looks like the last little pig was smart after all, until owning a nice brick home in a town of wooden houses all broken up with gas lines open all over the place starts to seem like a really bad idea.
This kind of Scorched Earth vision was all the rage in the last part of 1999. I was working in Financial Services doing portfolio analytics server-side in Java with a legacy back end of millions of lines of FORTRAN code. So we didn't sweat the Y2K bug at all -- financial analytics really have to be able to go way into the future for Bonds and what-ifs, so the system had always been written to handle Y2K dates. But I remember thinking that if the EastBayMUD servers, which I had toured, and were highly COBOLized, were to crash, then for a few days we might not have water. Though more likely was just an EBMUD water bill for a century's worth of water.
So why don't houses fall down when they issue a new nail? Because objects in the real world are manufactured, and then installed and left there with no dependencies. Having a build system is a luxury deployed by just about every software shop, and so is having developers who use version control, test suites, etc. Yet, when we build software, we don't package everything up and ship appliances. We ship tiny bits of code that depend on lots of other code. And our bits of code were made by the interaction of thousands of other littler bits of code. And we send all our little pieces, with all their dependencies on pieces that we don't ship, out in their little boats. Sorry, lost the house metaphor for a boat reference. But I like the imagery of it.
In the dot-com binge, my brother and I started ToasterWorks.com and designed what is now called Cloud Computing. We designed, coded, networked, prototyped, and documented most of, say, what you can get from Amazon EC2. My brother worked on selling it to Dell, Compaq, and some large ISPs. We got really close, but we got beaten to market by several competitors.
I knew going into it that the ideas I had were the result of being in a computer geek zietgiest that probably included about 2 million really sharp minds. I knew there would be lots of others with identical startups, because appliances had already busted out, and linx had GUI terminal servers, so the concept of provisioning a blade at boot time with a networked version-controlled repository of boot images and large data mounts seemed like the way to go since it was clear that big iron would make some kind of comeback. I was also working then on designing web services so that all our virtualized boxes could have very simple duties and just network their messages back and forth. But like I said, that's no boast--lots of people were all scratching their heads and thinking about the future and how it might be manageable with all these dang Linux boxes stacked in the server room and all these dang software pieces.
So now we have Cloud Computing and we have Web Services. But we are still missing some of the things on my wish list.
Real packaging. You get the immutable bits that will work. All of them.
You can still have a build system for assembling and testing these beasts. But don't expect users to have a build system that can handle your kruft.
Simple communication.
All text communication.
Bi-directional communication supported, and one-way messaging is preferred.
To implement this today, I'm thinking: XML messaging. XML file formats. Use schemae and attach them to the document. These all seem viable and established now.
Real interfaces. Typed, inheriting, globally known, versioned, assembled interfaces. (Possible with things like XML messaging and JSON. But also perhaps with sending objects around.)
Come up with some kind of object that can be sent around the network and stored and versioned and transformed, round-trip, to text and back. Java does this, and it plays well with real object databases. I suspect Javascript could do this well, too, as my brilliant friend Brian hipped me to, but he said he detests blog technology, so I guess I shouldn't link to him. Anyway, he says it's totally doable, and folks are doing it.
But I haven't had too much time to see what is up in the javascript-as-object-nirvana world, since I am happy as a clam since I got the schema for the anarchia.us objects coded in Java and stored in db40 and accessed with the rudiments of Retro (my command-line framework) working on my laptop. Java objects. Simple. Not "Value Objects," but POJOs. Properties, collections, relationships--all store nicely in db40. And I'm not doing any kruft. No annotations, no JAXB, no mapping file, no injection, no nothing.
I'll put up the source very soon, and design docs. For now, please read the other posts in this blog to see where I'm hoping to go.
Showing posts with label diorama tagonomy. Show all posts
Showing posts with label diorama tagonomy. Show all posts
Friday, August 6, 2010
Wednesday, June 16, 2010
A day in the life in diorama
Let me tell you how I work.
I keep all my documents in a hierarchy of folders.
The folders are all named tersely, but human-readable and obvious.
I put files together that are in related projects.
I maintain this hierarchy of folders in CVS, which allows me to archive it, along with dated versions of everything I add or change in the hierarchy.
I buy time on a cloud server and run several processes, one of which stores my hierarchy, in sync, archived.
My data are protected by industry standard security which would be good enough protection against any except the highest level hackers with access to large compute clouds (e.g. the government).
OK. Everything above is factual, extant, and uses existing open source technology. Everything that follows in this post is a dream, combining how I work, and new, open source software to model and automate what I now do manually, and what I cannot currently do, but wish I could do.
In addition to this hierarchical, archiving, versioning, distributed file system, I maintain my metadata in a system called Diorama. Diorama stores all the information about my files, and maintains this as a context that can be searched, executed in, added to. I can choose to add new items to my own Diorama-run file system, or I can add new items that are stored out in the cloud, in which case Diorama simply stores the URL and any metadata that get associated with that document. I use the Diorama-run file system for text and small images, and remote storage for larger, binary files. You could use any file system you want, and just have Diorama maintain metadata and indices to documents only.
In addition to tags and categories, where you were in the context at the time of document status change, can be saved in the metadata as well (if you allow your context history to be saved in metadata or metadata links to your context history database). Diorama uses hierarchies of tags as categories, with paths, in "tagonomies", or, tag-based taxonomies. The taxonomies are only suggested, and there are so many aliases, mispellings, case-changes, that they cannot be said to be purely concrete taxonomies, since they don't root together in a particularly organized or distinct way, and in many ways are really graphs. But Diorama soldiers on through the aliasing process, until small set of commands and small set of paths are what I work with most of the time, and those paths and commands become familiar to me and to Diorama, so they become quicker to execute in the command shell.
Since I work mostly through Diorama, everything I do is stored in my personal command history database. It is indexed by time. Documents and remote processes can link to the exact time I executed something, and for API calls inbound, any Diorama context I was logged into can always return its local timestamp, so that the history data are kept local, but can be indexed by request in the future.
Any time I am using Diorama, I get to keep history. So this works regardless of where I log in from. The history data are kept in my Diorama distributed database.
The Diorama command line is simply a command line environment that has a display area that can show menus of commands, pictures, web pages, etc. Diorama has a context area that shows information about the context you are in. Menus, commands, resources, shortcuts, notes, etc. You execute commands to open documents, take actions like send mail, create content, publish, and subscribe.
The context area shows you the tagonomy of where you are, and also the tagonomies of any document you have open, etc. So for this page I might see
"docs/dioramaBlog" or "projects/diorama/blogs/InThroughTheDioram". You can tag anything with these path tags (the character I use is a / but you could use anything, such as "projects :: blogs :: In through the diorama"). The system treats things in the path as parent contexts, and learns about how you categorize things this way. It starts suggesting context paths, for example. The system also does quite a bit of aliasing, phonic checking, and case-insensitivity.
Diorama also has a context area that shows you where you are in the web of contexts. Most of the contexts you will spend time in are yours, like your home directory, or directories of projects you are working on. But you stay in these contexts in multiple windows, and can switch contexts with the keyboard, while still in command mode on the command line. It's very much like being in emacs, sending mail. But it is very menu driven and simple, obvious, and fast. But you will be able to visit contexts of others. You will be invited to join these contexts, because a person on the other end knows you somehow. The system is based, just like human interaction in the offline world, on trust. Most people don't trust pimps, hookers, or car salesmen, but in fact these people are at least honest about their game. Still, when you go to hang out in the park with your best girl, you don't want pimps there, and when you go have a drink with your friends, you don't want a used car salesman standing over your table like a drunken mariachi. So Diorama has no ads. And everyone you run into, or whom you allow access to, is known to you.
I keep all my documents in a hierarchy of folders.
The folders are all named tersely, but human-readable and obvious.
I put files together that are in related projects.
I maintain this hierarchy of folders in CVS, which allows me to archive it, along with dated versions of everything I add or change in the hierarchy.
I buy time on a cloud server and run several processes, one of which stores my hierarchy, in sync, archived.
My data are protected by industry standard security which would be good enough protection against any except the highest level hackers with access to large compute clouds (e.g. the government).
OK. Everything above is factual, extant, and uses existing open source technology. Everything that follows in this post is a dream, combining how I work, and new, open source software to model and automate what I now do manually, and what I cannot currently do, but wish I could do.
In addition to this hierarchical, archiving, versioning, distributed file system, I maintain my metadata in a system called Diorama. Diorama stores all the information about my files, and maintains this as a context that can be searched, executed in, added to. I can choose to add new items to my own Diorama-run file system, or I can add new items that are stored out in the cloud, in which case Diorama simply stores the URL and any metadata that get associated with that document. I use the Diorama-run file system for text and small images, and remote storage for larger, binary files. You could use any file system you want, and just have Diorama maintain metadata and indices to documents only.
In addition to tags and categories, where you were in the context at the time of document status change, can be saved in the metadata as well (if you allow your context history to be saved in metadata or metadata links to your context history database). Diorama uses hierarchies of tags as categories, with paths, in "tagonomies", or, tag-based taxonomies. The taxonomies are only suggested, and there are so many aliases, mispellings, case-changes, that they cannot be said to be purely concrete taxonomies, since they don't root together in a particularly organized or distinct way, and in many ways are really graphs. But Diorama soldiers on through the aliasing process, until small set of commands and small set of paths are what I work with most of the time, and those paths and commands become familiar to me and to Diorama, so they become quicker to execute in the command shell.
Since I work mostly through Diorama, everything I do is stored in my personal command history database. It is indexed by time. Documents and remote processes can link to the exact time I executed something, and for API calls inbound, any Diorama context I was logged into can always return its local timestamp, so that the history data are kept local, but can be indexed by request in the future.
Any time I am using Diorama, I get to keep history. So this works regardless of where I log in from. The history data are kept in my Diorama distributed database.
The Diorama command line is simply a command line environment that has a display area that can show menus of commands, pictures, web pages, etc. Diorama has a context area that shows information about the context you are in. Menus, commands, resources, shortcuts, notes, etc. You execute commands to open documents, take actions like send mail, create content, publish, and subscribe.
The context area shows you the tagonomy of where you are, and also the tagonomies of any document you have open, etc. So for this page I might see
"docs/dioramaBlog" or "projects/diorama/blogs/InThroughTheDioram". You can tag anything with these path tags (the character I use is a / but you could use anything, such as "projects :: blogs :: In through the diorama"). The system treats things in the path as parent contexts, and learns about how you categorize things this way. It starts suggesting context paths, for example. The system also does quite a bit of aliasing, phonic checking, and case-insensitivity.
Diorama also has a context area that shows you where you are in the web of contexts. Most of the contexts you will spend time in are yours, like your home directory, or directories of projects you are working on. But you stay in these contexts in multiple windows, and can switch contexts with the keyboard, while still in command mode on the command line. It's very much like being in emacs, sending mail. But it is very menu driven and simple, obvious, and fast. But you will be able to visit contexts of others. You will be invited to join these contexts, because a person on the other end knows you somehow. The system is based, just like human interaction in the offline world, on trust. Most people don't trust pimps, hookers, or car salesmen, but in fact these people are at least honest about their game. Still, when you go to hang out in the park with your best girl, you don't want pimps there, and when you go have a drink with your friends, you don't want a used car salesman standing over your table like a drunken mariachi. So Diorama has no ads. And everyone you run into, or whom you allow access to, is known to you.
Subscribe to:
Posts (Atom)