Reddit as a Project Community

That's a few years since I've left this blog. You can now find more fresh news about the Entrelacs Project on the dedicated Reddit Community.

New documentation about Entrelacs

May I introduce you few presentations I've authored about the Arrows Space algorithms and architecture ?

• Arrows Space - Introduction
• Arrows Space - Tricks
• Arrows Space - Connectivity
• Arrows Space - Persistence Layer

These presentations aim to make the source code of the Entrelacs project a bit more understandable.

Entrelacs first prototype starts working

During this year, I did my best finalizing the Entrelacs prototype I started few times ago. It has been totally rewritten,  now with a 64 bits cell structure which may hold a whole arrow definition, ie. a pair of 24 bits adresses. It also includes a commit journal and a working functional machine which illustrates  how I imagine a self-efficient system. This code is now runnable and successfully passes few regression tests.

I've also completly rewritten the project introduction into the form of a manifesto. My plan is to separate and highlight the core ideas I believe in from the code I've authorised. By doing so, I hope to make the Entrelacs project an umbrella project working as a reference for people wishing to explore the paths I went into. I' m nothing like a leader but necessities push me to act so.

Here are my next plans.  On the theoretical side,  among other things, I plan to work on a peer-to-peer scalable architecture of networked entrelacs system. On the political side, I plan to introduce Entrelacs as a easily self-hostable server of cloud-like applications which may counterbalance corporate-driven cloud services. On the coding front, I obviously plan to embed an http server in my prototype so to serve arrows as REST resources. Of course, it has surprising implications, like merging resources and url concepts into a single one.

All in all, I'm bound to merge the two projects I'm working on. That is Handballer and Entrelacs. My Entrelacs prototype will be an http server publishing the Entrelacs language interpreter I'm working on. But its key feature will be to allow user agents to register a "closure" as an url that  the server will call back on invocation. So Handballer will be operated here to open http push connections and convert them into reliable url. This duo will transform Entrelacs into a Restful applications bus, that is a new form of application server to host complex AJAX/COMET web appliances in a very open and transparent way.

The first Entrelacs prototype is emerging

I've eventually decided to build up the first prototype of the Entrelacs system. After a few weeks of work, I think I finished these last days writing the first and main software component, that is the arrows space.

The arrows space is a kind of data base which allows to store and fetch information in the form of arrow structures, in compliance with the arrow paradigm.

As I expected, the resulting piece of code is very short. But it isn't as simple as I wished.

Of course, as predicted by the arrow theory, the mass memory stands as a self-referred open-addressed hash table using all the available storage space.

But I had to better integrate data strings in my plans.

A data string unambiguously defines a tag arrow. I decided a long time ago that such data would be stored among the space cells (hash-table buckets). However, the point I missed is that such a data sequence can't be mixed together with data already present in expected locations by following the open addressing way. One must link all string slices each other with a kind of "next" field.

But I didn't want to use a full memory pointer to link related cells. It would have double the storage need. So I tried to be smart and decided to introduce a tertiary hash.

As a reminder, the "double addressing" approach consists in computing 2 hash codes. The primary hash gives the default cell where to store an arrow while the secondary hash gives a probing offset which is used to look for a vacant cell when the primary cell is already used. Using these two hashes theoretically avoid data clustering

I decided to introduce a tertiary hash so to compute a new offset: the sequence offset, which separate cells involved into data string storage.

So upon conflict, one only needs for a few administrative bits per cell to build an offset "factor". This factor stands as a replacement for a "pointer". If there is no unwanted data between a cell and the next cell in the sequence, this factor is set to one. But when conflicts occur, this factor allows to jump over all unwanted cells up to the next cell included in the wanted sequence. Of course, this value range is very short but one considers that many consecutive busy cells is rare.

I found this solution so handy I eventually decided to use it for arrow back-references list storage (even if such lists can be safely polluted by unrelated data).

However, I had to made concessions.

• BLOB arrows raw data are not stored into the arrows space. They are stored as host files instead. As long as Entrelacs won't be an autonomous operating system, I guess this is a reasonable concession.
• Incoming children arrows and outgoing children arrows are put in the same back-references list. It means one can't distinguish them without fetching them and examining their ends. This is a bit disappointing news, but I can't have more than one coalesced linked list per arrow.
• For the same reason, tag arrows are not back-referred. Otherwise, the back-references list would have been put beside the char string in the same coalesced linked list. Tag arrows only have a reference counters used for garbage collector.

At first, I intended to reuse the factor bits to build up a coalesced list of all arrows sharing the same primary offset by defining factor*probe-offset jumps. It would have lead to a whole coalesced hash table enhanced by a double hashing trick.

But for the first prototype, I had only 8 admin bits per cell. Combined with 24 bits for a memory address, it just fits in a 32 bits cell.

Now, I use these administrative bits for two other needs.

• to distinguish between regular arrows, rooted arrows, tag arrows and stuffed cells.
• to count cell "crossovers". This other invention of mine consists in incrementing a short-range counter per cell each time the cell is included into a probe sequence. It is supposed to make cell freeing feasible as in linear probing. TBC!

So I was obliged to reuse the same bits to store either the offset factor or a cell type, depending on whether the cell is an arrow definition start or some of its subsequent cells. It was hard but I finally found a working use of these 8 administrative bits.

I've thought about this code for like 11 years. So I'm more than happy to see something concrete emerging. Now I must confess I need a break. I haven't even tried yet to compile or obviously to debug the whole thing. It also lacks for a few features, like a commit log to ensure transaction atomicity.

Then the next step will be to code the virtual machine for the Entrelacs language -a query language for the arrows space-. Sigh! If only I could be seconded in this task.

Misunderstood Chrome OS

At first, I though the false beliefs about the new Google's OS were due to a temporary misreading of Google's plans.

But at the moment writing, even the rare IT experts who positively welcomed Chrome OS persist in propagating even more wrong beliefs about it. To name a few:

• Chrome OS is a "crippled OS" / a "toy OS" / a "secondary OS"
• Which could NOT run Photoshop or WoW
  
• Which depends on a network connection to run
• Which can't compete with regular OS for non Internet centric use cases.

Chrome OS is not more dependent to internet than any recent OS.

To make a comparison, the computer I used to write this blog combines regularly updated binary Linux packages. As a whole, one may see this software as a local mirror of a remote Debian repository. My firefox browser may also been seen as a local copy of some Mozilla's ressources.

On the other side, Gears and now HTML 5 allow a web application to be fully mirrored and runnable in a local storage.

Is there really differences? all these software are updated when a connection is available, except than web based technologies make this process more open (any domain can be a software provider), more secured (same domain confinement), more structured and fine-grained (data base) and generalize it to both uploading and downloading and both application code and user data.

Please note Gears is already a several years old technology, now superseded by HTML 5 local storage features. Both Gears and HTML 5 are fully integrated to Chrome browser. It means nothing prevents software developers to design applications 100% featured offline.

I think one doesn't even need a network connection to install such a web application as I suppose you could install it from a "file:" URI pointing e.g. a plugged SD card. You understood well: Not only Chrome OS applications could run without a net connection but they could even be "installed" offline.

Chrome OS can run anything.

What a web application will be able to do at the date Chrome is available? Well, as a matter of fact: _everything_. Thanks to HTML5 last additions, the Canvas element,  JIT compiled Javascript,  O3D, and last but not least the very promizing NativeClient concept, any existing desktop application can be ported as a web app.

• For example, your preferred MMORPG could be replaced by  as an O3D/JS app, seconded by a NativeClient framed IA and physic engine.
• People often cite Photoshop as THE application which will never be available on Chrome OS. On my own, I don't see why it could not be cloned as well. Simply combine O3D, Javascript and NativeClient framed image processing native code.

Chrome OS is not a crippled OS.

According to me, Google is achieving the most consistent, secure and easy to hack applicative framework ever made, while being lighter and as much efficient than any classical software stack.

Google is not (yet) like Microsoft but...

Saying "Google is like Microsoft" gets more and more trandy. But remember how open Google API are. Standard API are always privileged by Google even when they conflict with a former Google's proposals (HTML5 vs. Gears). Most of the Chrome OS software is released as Free Open Source Software under the Chromium brand-neutral project.

Even if Chrome OS also features a form of trusted computing, I'm confident with the fact that the end user will be able to bypass security checks and even install an alternative OS. Why? Because Google is neither a software nor an hardware authoring company. It's a SAS provider whose long-term strategy consists in removing all the mess between their services and their users, especially closed software on mobile and wired devices.

However I'm OK complaining about how Google is centralizing and mining our user data. As long as Google deals with ads, no problem. But at any time the Google's architecture could be high-jacked by some governments or corporations to make 1984's world a truth.  This is clearly a "single point of failure" in human rights protection.

My idea is that we can make personal data accessible from anywhere without storing it "on the cloud". The solution consists in self-hosting SAS at home with low-consumption hardware. This is the necessary condition to sanitize this new software architecture Google has built for us. It pleads for a complementary software stack providing self-hosted services at home. HandBaller and Entrelacs could contribute to it.

Nickel OS: a home server for Chrome OS and relatives

When I coded Handballer few years ago, I made the assumption new kind of devices would come along, featuring a full open-standard browser on top of an extra-simplified Linux distribution. I was inspired by preexisting gadgets like the Sony PSP, Opera DS, few costly MID or even the Nokia N770 Internet Tablet.

However, new kind of devices came along with new powerfull low-consumption chipsets and WebKit based browsers: the iPhone, the Palm Pre, Android phones... They all can run RIA in quite a pleasant way.

Today Google releases Chrome OS design docs and source codes, materializing a new step forward this direction. And even if most people still don't understand the point of this OS, it achieves the transition to a new software architecture, promoting the web engine as the converging head of every strategical software stacks: graphics, input, networking, data persistency, applications run-time, and so on.

Combined with latest HTML and Google technologies like Gears and Native Client, the browser is bound to be the structural frame of any existing application both in on-line and off-line mode: Gaming, Multi-media and Office Functions.

However Google is not exactly a non-profit organisation. Many people around the world are wondering about the Google model: hyper-centralization of the user data within big corporate owned servers. Is it really sane? Is there a real user benefit? Is it KISS?

On my own, I suppose FTTH won't come up to my home before decades. I could even lose my Internet access in the future. Who knows? But most important is that there's plenty of daily use cases which don't need to deal with "cloud" located services whatever the bandwith. Period.

So what's the best thing to do right now. Return to the old clumsy Linux/Mac/Win platform? No way. The house-cleaning that Google did with Chrome OS is IMHO excellent. Its architecture is so simple and straight-forward I already know I won't be able any more to stand for a full minute just to switch on a web radio, type an instant message or watch today's weather.

Naaaa, the best thing to do now is to calm down Google fantasies by relocate some strategical user applications at home while leveraging on web technologies.

It won't need many watts to make user data came back to home. Hardware technologies are almost ready for this purpose. Two ridiculous watts should be soon more than enough. My own NSLU2 home server drawns only 4 watts even if it was designed 5 years ago.

However the FOSS community isn't prepared for the counter-offensive. We need a web-based application framework which will behave at home like the Google ecosystem does on the cloud, that is in a homogeneous and reactive way. My thoughs float around this central idea: we need for a HTTP middleware within our home LAN to connect all our home-based resources each other and publish them back to a web terminal.

Think about an oversimplified Linux embeding an light HTTP server connected with home or internet services. What Google did for a personnal web terminal has to be done for a personnal web server.

HandBaller is probably not the definitive solution to build such an home HTTP server but it shows what we need for: A HTTP software bus melting an AJAX router and a COMET server to make home computing resources collaborating together for the user needs.

Imagine what kind of Home "Mash-Up" you could build from such a framework, like browsing your media storage then by a single web-click sending a self-made video to your networked TV while syncing your facebook status at once. What if the same thing could be done remotely in a perfectly transparent way through any web terminal connected to your home IP address.

We don't need for Google nor any ISP platforms, software or hardware to do so. We just need for the server counterpart of Chrome OS. Since Nickel-Chrome is such a famous couple, we should even call it Nickel OS.

See you soon.

HandBaller project blog opened

[EDIT]Now, this blog will officially give news about the Entrelacs project as well. For more information, please visit the Entrelacs projet page.[/EDIT]

Welcome to HandBaller project blog. This blog will gather useful information about the project, its status and future. For more information about this software, please visit the HandBaller project page.

This blog will also share some pretentious thoughs about data-computing paradigms and such. So please stay tuned.