Some time back, Melanie added a new flag to items, the “Export” flag, which some viewers started to support. The Export flag allows creators of content to mark them as exportable or not exportable — meaning ok to export to other grids or not. That gives some level of flexibility to content creators, but it doesn’t solve the problem. If an item is marked “No copy” but “Exportable”, a bad grid operator can easily change the permissions on it. So, at the moment, our recommendation to viewer developers is that they only allow to mark the “Export” flag when the item is fully “free”: no constraints on copy, transfers, and modifications. This is so people don’t get a wrong sense of security.

I should note that there are no reports of Hypergrid thefts, even though they are technically possible. As far as I know, most copy-botted content comes from using altered viewers on closed, non-HG grids; those loots are then imported into other OpenSim-based grids via those viewers too. So altered viewers, not HG thefts, are the main problem. Nevertheless, the fact that the HG opens additional doors for abuse has been a deterrent for commerce on the Hypergrid, so much so that we recommend “Export” only for freebies. It would be nice to change that perception. The incomplete proposal I have here solves the problem not just for the Hypergrid but also for viewer-based thefts in closed grids, even if those grids don’t ever want to connect to the Hypergrid. It solves the problem so thoroughly that, if implemented, it would allow commerce on the Hypergrid to bloom. The problem is in the implementation; hence, the “incomplete” in the title of this post…

The proposal, in a nutshell: ledgers.
Specifically for the Hypergrid: distributed, decentralized ledgers. Yes, blockchain, but minus the coin.

How it works, birds-eye view

Let’s ignore implementation details for a moment. I’ll cover them at the end.

Here is the main idea of a ledger: every time an item changes hands (from one avatar to another) a record about the transaction should be entered on a database. The information should include: the item ID, the time of original creation of the item, the timestamp of the transaction, the owner ID, the receiver ID, and the item’s permissions at the time of the transaction. Ledgers are append-only databases: they hold the historical record of all transactions since the beginning of time. (A side note to mention that OpenSim already has append-only databases: the assets.) So if any item’s origin is ever in question, the ledger can be queried for the ground truth.

Next, I will go through various theft scenarios, and explain how the ledger gives honest grid operators the technical justification for punitive action against thieves, and how it it will help identify (and, hopefully, socially isolate) dishonest grid operators.

Detecting theft within a closed grid

Let’s ignore the HG for a moment, and focus on centralized ledgers.

Let’s consider the viewer-based theft scenario in closed grids: some bad actor logs in with an altered viewer and copies all content that it sees into this bad actor’s inventory, changing permissions, and even authorship, along the way. This bad actor goes on to sell this content as their own on that same grid. Nothing in this proposal prevents this from happening. However, if and when the original creator detects this shenanigan, they can file a complain with the grid, who now is armed with the ledger, the historical record of all transactions. A simple query on the ledger can find out when the items in question were first transactioned; the offender’s item’s timestamp will be later than the original creator’s item’s timestamp. If the items are, indeed, similar, as per the original creator’s complaint, then the timestamps, and the lack of a transaction chain between the items, are proof that there was a theft, and who the offender was. The grid owner can act swiftly.

Detecting theft across grids

Let’s continue to ignore HG for a little longer.

Let’s consider the same viewer-based theft scenario, but now the bad actor imports the stolen content from closed grid X into another closed grid Y via the altered viewer. Assume both grids have their own separate ledgers, and both grid operators are honest. Again, this proposal doesn’t prevent theft from happening. It’s only when suspicion of theft is raised that the matter can be investigated.

In this scenario, suppose the bad actor with the altered viewer imports the stolen content into grid Y as their own. Grid Y will store the item as if it was legitimately created by this person at time T1, when it was imported there. It knows nothing about its true origin. Some time later, there is a complain from the original creator who first created this item in grid X at time T0 < T1. Both grids can easily investigate by agreeing that the items are, indeed, very similar, and by checking their own ledgers independently. If they find that T0 < T1, then it’s clear theft was at play, and who the thief was. They can act.

Detecting theft between a closed grid and OSGrid

Continuing to ignore HG, let’s focus on hybrids like OSG: in OSG-like grids, many independent people control their regions, but they are all connected to central grid services operated by osgrid.org, specifically inventory; this ledger would be another grid-wide service. Assume the region operators may be bad actors, but that the grid services operators are honest.

In this scenario, a bad actor, armed with an altered viewer, logs in to a closed commercial grid, steals content via the viewer, and then imports it to OSG using their own region, where they not only have God-like powers but can also alter the simulator code. They have a lot more opportunities to tamper with the timestamps of creation, ownership, etc. But that doesn’t matter much, as long as inventory and ledger are centralized services of the grid, and are capable of correcting any region-sent information. When suspicion of theft is raised, both grids can check their ledgers for timestamps.

Managing transactions across the Hypergrid

Having separate ledgers is fine for grids that trust each other, but in the Hypergrid, some grid operators may be bad actors themselves, and they control their grid’s ledgers. If they steal something, they can then enter those items on their own databases as having been created in a date that is sufficiently in the past to escape proof of theft, or, even worse, to harass the original content creators! “I created this before you did!” That’s bad.

For this to work on the Hypergrid, we need to establish a distributed, decentralized ledger system, blockchain style. Here is how it works.

Every time an item changes hands (from one avatar to another) a record about the transaction should be entered onto the distributed ledger. The distributed ledger is a kind of replicated database over a peer-to-peer network, but where peers can independently verify the veracity of the information that is sent to them — sometimes directly and sometimes via some voting algorithm. If we trust the information on the distributed, replicated ledger, rather than on any single node, then we can always verify whether something was stolen or not across grids that are part of the network.

The weak link of decentralized ledgers is false information. How can we prevent false information from entering the ledger? Here are a couple of scenarios of false information:

• Problem: a bad grid operator creates a false record of a transaction between a visiting avatar and a local avatar. That is, they send to the ledger “HG visitor A gave a copy of item H to local avatar B at time T1″ when that transaction never happened.
  Solution: any transactions between avatars should always take place on the grid of origin of the avatar that gives the item. In other words, visitors can get things, but should not give things. All transactions should be local to the originator avatar. This can easily be checked at ledger-entry time by verifying that the issuer grid is the grid of the originator avatar. If it isn’t, nodes of the distributed ledger won’t accept the transaction.
  (We can implement non-local transactions, too, but that will complicate the giving procedure substantially, as it will require confirmation from the originator when it comes back to their grid)
• Problem: a bad grid operator changes the items’ creation time to the past. For example, they import stolen content with an altered viewer, and then they send the following record to the ledger “Item H was created in 12/12/2012.”
  Solution: ledger nodes need to verify that the timestamps of a creation record are credible.

Implementation Issues: Where Things May Come to a Halt

While the general idea will definitely be the best solution for protecting content, implementing it in practice for the Hypergrid is not a trivial matter.

Ledgers are append-only databases; their size can grow very large. This is a common problem of all blockchain solutions. For example, the Bitcoin ledger currently 275G, and it will continue to increase forever. In our case, we could either do some garbage collection when items are removed from inventory, or we could do what blockchain operators do: archive old records into separate, slower, databases. But these solutions aren’t very good for the Hypergrid, where a large number of single person grids exist, many of them running in people’s home networks. Those small shops would end up having to deal with a very large database, most of which is irrelevant to them; they would get it just because they would be part of this ledger network.

We can estimate how large a distributed ledger for the HG would be, and how fast it would grow based on the current estimates of people’s inventories. One thing that will definitely contribute to the large size of the ledger is if we enter every inventory item on the ledger: if anything created in inventory, whether it will be passed around or not, is entered into the ledger, the ledger will become unsustainably large very quickly. For reference, there are hundreds of millions of entries in OSGrid’s inventory service, and probably billions of items have been created there. Multiply that by 1000 grids. This would quickly grow to be a terabyte-size database.

We could focus only on the items that are transferred from one avatar to another, and use the timestamp of the first transaction as a proxy for time of creation. This approximation will fail to detect thefts if the item is stolen before the first transaction happens, and then the thief quickly makes a first transaction themselves. This should be rare, and people could be aware and compensate for it. It will definitely decrease the size of the ledger by several orders of magnitude, as the vast majority of items in people’s inventories aren’t passed to another avatar.

Still, even with optimizations, the ledger will be large, and this will be a problem for small standalone grids.

Perhaps the right thing to do is to decouple the ledger from the grids. We can implement a distributed ledger system that is operated by a few trusted people in the community, with access to enough resources, and that ledger can hold the transactions of everyone who wants to ensure the use of legitimate content in their grids. Any grid not on this ledger will be suspicious; social pressure will make them look bad.

Of course, nothing prevents a single entity from creating a centralized ledger for many independent grids, as a service. That will require complete trust in the ledger operator.

Anyway, I hope this post will trigger some discussion and maybe even some action, as it is clear that without a mechanism for content protection, commerce on the Hypergrid will never take off, and the nice professionally-created items will not be available to those with good intentions, such as Universities.

It’s been a while since I last posted about HG 2, but it’s finally coming to life! HG 2 is all about letting grid operators define policies for access control to their grid’s assets and users. I thought I’d show how it is taking shape, so that you’ll be ready for it in the next OpenSim release.

HG Inventory

Inventory is much more secure than in HG 1.5, and the My Suitcase folder is playing an even more important role in that.

The first difference you will notice is that this folder is now under the root folder — see picture on the left. In HG 1.5, My Suitcase was at the same level as the root folder. That posed some issues, because the viewers can’t really handle that. So we had to make up Web applications (like D2 Wifi) to make inventory manipulations between the Suitcase and the other folders. Kind of hacky. That’s not happening anymore. In HG 2, My Suitcase is inside your normal inventory tree, so you can easily copy items back and forth between that folder and others.

Like in HG 1.5, the My Suitcase folder is special: it is the folder tree that receives objects you collect while you are visiting other grids. But now it is even more special: it is the only folder tree that is accessible to you (and therefore to the rest of the Internet) while you are traveling. Period.

Look at how your inventory will look like in HG 2 when you are outside your home world — see picture on the left. As the picture shows, all your folders except My Suitcase are shown as (Unavailable). You can still see them, and unfold them. You may even be able to access items that are already cached in your viewer’s cache (textures, notecards, etc.). But if you try to access any items outside My Suitcase that aren’t cached in your viewer while you are outside your home world, the operation will fail. For example, rezzing objects from those folders will fail.

Items inside My Suitcase, however, will be available. My Suitcase will also be the tree where all the items collected while traveling  will go. Once you are back home you can then move them into other folders.

In short: the Suitcase is now really the only thing you’ll have available while traveling (but keep on reading for some exceptions). That means that it is also only the only thing that is available for rogue grids to mess with. Treat it as your own real-world suitcase: it may be stolen or badly damaged. But the rest of your belongings are safe in your home world.

[UPDATE 9/23] I should be more clear. The existing HG1.5 inventory service, which lets users still access their entire inventory abroad, will still be available in the future. Grids can continue to use that. They can even use the normal inventory service (HG1.0 style) on the Hypergrid, which has no restrictions whatsoever about inventory manipulation. Both of these, however, especially the normal inventory service, are horribly insecure and expose users to all sorts of risks regarding their inventory. Unless the networked grids have 100% trust in each other, I don’t recommend using them.

Appearance

Appearance in SL/OpenSim is a very tricky thing. Your appearance as an avatar consists of two things: wearables (shape, clothes, etc.) and attachments (prim hair, etc.), which can be arbitrarily complex objects with textures, scripts, etc. Additionally, the viewer “bakes” all the wearables into a set of textures, and those are the ones that are actually passed around between your viewer and the viewers of all other people around you. To make things even more complicated, the attachments have a nasty dependency with your inventory — they are objects that must be somewhere in your inventory, and the viewer must know about them being attached to you.

Why am I telling this? So that you understand some of the challenges regarding inventory access and asset access control related to your appearance.

Your appearance’s assets will be copied to the worlds that you are visiting, even if the items that you are wearing are not in the Suitcase folder. If we didn’t do that, you’d always have to switch into an appearance whose items would be in your Suitcase, and that would be a pain. So, your appearance items (your clothes, body parts and attachments) are the only exceptions to the rule “the only items available outside are items under the My Suitcase folder.”  They are copied, and, as such, assume that they will be publicly available.

But this is not good enough for some grids. Suppose you are a user of a commercial grid wearing a very expensive avatar. That grid may not want any of its assets to be exported to other grids, including the avatar that you are wearing — that refusal of service can now be done (read on). But if you arrive at the destination without any wearables and attachments, you will be seen there as Ruth, even if you see yourself wearing the same expensive avatar because it is cached in your viewer as you travel. Projecting a Ruth image to others is not something we consider a good user experience, so we had to fix this somehow.

There is a configuration variable that grids can set regarding the use of “allowed appearances for Hypergriding.” That is, grids can create one or more appearances with assets that they don’t mind being publicly shared, and then force their users to wear one of those appearances before hypergriding out. The picture on the left shows the warning message when a user is wearing something that is not part of the allowed HG appearance assets, in this case, it was the sphere attachment.

Unfortunately, due the lack of appropriate viewer features, the appearance switch has to be done explicitly by the user, it can’t be done automatically. So if the user gets this warning message when she’s trying to HG TP, she needs to replace her outfit with one of the official ones. In the case in the picture, it’s a folder called HGOutfit.

 Assets Access Control

The access to a grid’s assets in the Hypergrid is done via a service called the HG Asset Service. Grid owners can now configure that service by specifying policies for exporting their grid’s assets to the outside world. For example, they can say that it’s ok for others to access textures and sounds but not scripts or meshes, or any other combination — the policy specification language takes the entire set of asset types. Grid owners can also specify which kinds of assets the grid can import from other grids.

The specification is done under the section [HGAssetService], something like this:

[HGAssetService]
    HomeURI = "http://127.0.0.1:9000"
    ;; The asset types that will never be exported
    DisallowExport ="LSLText, LSLBytecode"
    ;; The asset types that will never be imported
    DisallowImport ="LSLBytecode"

When these policies are specified, those types of assets will never be imported/exported, no matter what users and the rest of the world may do.

Going one step deeper into content protection, commercial grids with protected content may want to physically separate their internal asset server from their HG asset server, i.e. they can operate over completely different databases. Their internal asset server can continue to be behind the firewall. With the Robust architecture, this has been possible for a while. But now it is possible to make that physical separation meaningfully regarding the user’s Suitcase folder. When that separation is in place, the assets for the Suitcase items will be stored in the HG asset server database, and users cannot copy items back and forth between the Suitcase and the other folders. The items in the Suitcase folder will be  unavailable in the users’ home world, and become available only when the user is traveling. This may be a desired feature, as commercial grids may not like the import of freebies from the outside. This requires some advanced configuration, but that’s the kind of security barriers that need to be in place for commercial grids to start experimenting with the HG.

For the time being, asset access control is done in bulk, based on asset types, and specified by the grid operator. There is no fine-grain asset access control [yet]. What this means is that grid users cannot express their own policies regarding their own objects, they are at the mercy of the grid operator. Bringing that specification up to the users will require some User Interface work (possibly on the viewers) that isn’t done yet. We’ll leave that for HG 2.5.

Users Access Control

In an HG Teleport, there are always two independent authorities involved: the home world of the user and the destination world. These can now be configured, also independently, regarding HG travels. It is now possible to control who goes where. This is the basis for the emergence of HG sub-networks — groups of grids connected to each other, but not to everyone, via the HG.

Incoming HG visitors

The control of which kinds of users can visit our world is done in a service called the Gatekeeper. It is now possible to specify whether the grid accepts foreign users or not, and, if it does, from which domains those users should be. When a user tries to HG TP to a grid in which she is not allowed, she gets a message like the one seen on the picture on the left “Destination does not allow users from your world.”

The configuration looks like this:

[GatekeeperService]
...
  ; ForeignAgentsAllowed = true
  ;;

  ;; If ForeignAgentsAllowed is true, make exceptions using AllowExcept.
  ;; Leave blank or commented for no exceptions.
  AllowExcept = "http://griefer.com:8002, http://enemy.com:8002"
  ;;
  ;; If ForeignAgentsAllowed is false, make exceptions using DisallowExcept
  ;; Leave blank or commented for no exceptions.
  ; DisallowExcept = "http://myfriendgrid.com:8002, http://myboss.com:8002"

The fine-grain control of specific users can continue to be done with ban lists and other mechanisms that already exist. The kind of control we added in HG 2.0 is domain-based, so all users from those domains will / will not be allowed in. That is what allows the emergence of HG sub-networks.

Outgoing local users

Conversely, the home world of users also has a say about where the users can and cannot go. That is done in a service called User Agents. When a user tries to HG TP to a grid which her home world doesn’t want her to go, she gets a message like the one seen on the picture on the left “Your world does not allow you to visit the destination.”

On this side of things, the policy specifications are based on the user-level. That is, users at different levels may be subjected to different restrictions.

The configuration looks like this:

[UserAgentService]
...

;; Are local users allowed to visit other grids?

;; What user level? Use variables of this forrm:
 ;; ForeignTripsAllowed_Level_<UserLevel> = true | false
 ;; For example:
 ForeignTripsAllowed_Level_0 = false
 ForeignTripsAllowed_Level_10 = true ; true is default
 ;;
 ;; If ForeignTripsAllowed is true, make exceptions using AllowExcept.
 ;; Leave blank or commented for no exceptions.
 AllowExcept_Level_10 = "http://griefer.com:8002, http://enemy.com:8002"
 ;;
 ;; If ForeignTripsAllowed is false, make exceptions using DisallowExcept
 ;; Leave blank or commented for no exceptions.
 DisallowExcept_Level_0 = "http://myfriendgrid.com:8002, http://myboss.com:8002"

I usually don’t write about the future, I prefer to write about things that are already done and working. But I’m going to make an exception, because people are starting to ask about configurations of the Hypergrid that pertain to the next level; we’re almost there, but not yet. I am going to give an overview of what’s in the works right now that will result in the next big bump in the HG number: the big two-point-oh.

As usual, some context first.

The very first version of the Hypergrid, HG 1.0, was done back at the end of 2008, and continued through 2009 and 2010. That first version was just an exercise in feasibility and in interacting with the opaque, black box viewer. Could I trick the viewer to think that it was still on the same grid while in reality sending the user to a different grid? — that was the humble goal of HG 1.0. HG 1.0 was not concerned with security at all, as the main goal was to figure out how to use the viewer in ways for which it had not be designed. Once I mastered the signaling of agent teleports, it was time to take the HG to next level: more security all around.

The next version was HG 1.5. The conceptual work for it started in early 2009, although the actual coding only happened much later. This was because in order to implement the Hypergrid as a collection of additional, optional services that don’t interfere with the simplest configuration, OpenSimulator itself had to be considerably refactored. That took a long time. HG1.5 came with OpenSim 0.7 in July of 2010. It featured identity security as well as a semi-secure (or semi-insecure, depending on whether you are an optimism or a pessimist) Hypergrid inventory. It also had all the right architecture for enabling HG Instant Message and Friends. That’s where we are right now.

So let me explain what we’re working on for HG 2.0

Inventory

Hypergrid Inventory is going to be 100% secure and the Suitcase will finally be usable without external Web apps. This required some heavy hacking in order to understand how to further trick the viewer into submission (i.e. use it beyond its specs), but we did it. Here is what will happen. When a user goes out, her entire home inventory tree will be removed from the viewer and replaced with the Suitcase folder tree only. That’s the only portion of the user’s inventory that will be exposed to the outside and to the user while the user is hypergriding. That way, the user’s home inventory will be out of reach and completely protected. Once the user returns to the home grid, the Suitcase wannabe-root folder is replaced by the user’s real inventory tree again.

Depending on configuration, the Suitcase folder may or may not be available to the user when she is in her home grid. When it is available, it will be a regular folder under “My Inventory”; the user can then move stuff from/to the Suitcase to/from other folders using the viewer.  But some grids may not want their users to bring in stuff from the outside, they want to continue to segregate content, so availability of the Suitcase folder in the home grid will be configurable.

Appearance

The appearance that a traveling user has may or may not be the same as the appearance that she had when she left the home grid. Again, this will be configurable. But the idea here is that  some grids don’t want to allow the export of appearance assets at all, so they may provide generic appearances to their traveling users, and dress them with their appearances when they come back. Other grids may not care about this, and appearance will be preserved.

User Access Control

There will be a means to specify several policies of access control: who can go out and where, who can come in and from where. These policies are based on several pieces of information about the users themselves and about their grids of origin.

So for example, it will be possible to specify that only users of a certain level can Hypergrid out, while others under that level can’t (a requirement often heard in educational settings). It will also be possible to specify a limited set of grids that the users can visit; or a set of grids that the users cannot visit.

On the receiving end, it will be possible to specify only a limited set of grids whose users are allowed in; or a set of grids whose users are not allowed in; or no foreigners allowed in at all.

Assets Access Control

There will be a means to specify whether the grid allows asset exports to other grids, and which grids, as a whole. HG 2.0 will not support the specification of fine-grained asset exports, i.e. I can’t say that this particular set of assets can go out, while that other can’t. This will require a means for people to be able to express that, and we don’t have that means yet.

≈

What this amounts to is that the Hypergrid will be able to support different policies that better reflect different levels of comfort that people have regarding opening up their worlds. Grids that deal with protected content will be able to continue to protect that content while allowing their users to visit other grids, and even collect items there, and allowing foreign users to come in and mingle — without any asset ever being exported or imported from/into their main storage. Grids that deal with kids will be able to block them from hypergriding while allowing certain privileged staff to go out and get content and knowledge from other grids. Secretive grids will be able to prevent foreign visitors while allowing their users to go out. Grids can have only one or two regions available to foreigners, while the rest is off-limits. Communities can make up small networks of worlds that allow Hypergrid exchanges only among themselves. Etc.

I hope this clarifies where we’re going with the Hypergrid!

I know that the first question that everyone will ask is: when? My guess: sometime this summer, hopefully early.

There are many improvements in OpenSim, most notably the new LSL NPC stuff — this allows you to create and manipulate Non-Player Characters (bots) directly from scripts. Really cool!

Hypergrid-wise, the most noteworthy development is the support for Friends across the Hypergrid. As explained in a previous post, I have created a world-to-world social network platform in OpenSim! Diaspora, anyone? Well, I get to actually do it and deploy it in  your worlds

Have fun!

P.S. I also made a new release for Wifi that works with Robust 0.7.2. Enjoy!

[UPDATED 6/2/2011] Added pictures!

Here’s a base case scenario. You run your own OpenSimulator VW, and you visit OSGrid with an Hypergrid teleport. Once you get to OSGrid, you meet someone there — maybe from OSGrid, maybe from some other VW –, and you want to add that person to your list of friends. People in your friends list acquire certain rights over your information shadow, and vice-versa: they may be notified when you login/logout (and vice-versa), they may change your objects (and vice-versa), they may see you on the map (and vice-versa), etc. An entry on your list of friends is a persistent reference to a user, so you can easily IM them, see their profile, etc.

Up to now, it wasn’t possible to hold references to users in other services. With these changes, that has become possible. An HG friendship establishes 2 pairs of data, each stored in each user’s user services. That data is “signed” with a random number known only to the 2 services and the simulator where the friendship was established, so that, even if the user services (hg friends, to be precise) are exposed to the Internet, no one but the holders of the signature can manipulate the friendship data. Here lies warning #1: don’t make friendships in worlds that you don’t trust — instead, ask the person to come to your world, or vice-versa, and establish the friendship there.

Independent of that, HG friendships established while you are Hypergriding require an additional confirmation step once you return to your world. Here’s a typical interaction: you go elsewhere, you ask someone to be your friend (or vice-versa), the viewer will tell you that the friendship was established. When this is done within your world,  the interaction ends there. With HG friendships, once you return to your world, or login the next time, you will be once again prompted for confirmation of the friendship. This is so that your friends list won’t get spammed with bogus entries sent by malicious components. Once you confirm that second time in your home world, the friendship is finally established.

Your HG friends can, by default, see your logins and logouts of the virtual world — to some extent, and this is where things are still work in progress. So far, they will be notified if they are in their own worlds. However, they will not be notified if they are elsewhere — not yet, at least.

This starts to show the really interesting challenges of these social functions done in a decentralized architecture: you can “move,” that is, your client doesn’t have a communication channel with one single server over your entire session; instead, your client “moves” from server to server. Keeping track of where people are (that is, which server their clients are talking to) is a major challenge! — especially without central servers. But I’ll come to this further down when I explain IM over the Hypergrid.

Another interesting aspect of the design of friends over the HG is the rights to edit each others’ objects. Within one world, once you give someone the right to edit your objects, that right holds for all regions of that world indiscriminately. With the Hypergrid, we have the option to differentiate between the worlds, and give rights to edit objects per world. In other words, if we are friends and I build things in your world, I can give you the right to change my objects in your world; but I may very well not give you the right to change my objects in my world. (or vice-versa). Right changes affect only the world where the action takes place: if I change rights in my world, those new rights will be stored in my world only, they will not be disseminated. When you HG-TP to a new world, you will be notified of the rights that you currently have over your friends’ objects there, if different from those in your home world.

This is a subtle semantic change. I like the more expressive capabilities, but I understand that people coming from walled-garden worlds may be somewhat confused by these extra options. We’ll see how this design decision fares as we go along.

One thing that is still work in progress: rights to change objects in 3rd-party worlds.

Now… Instant message.

The base case is very simple: you are in your world, your friend is in her world; you IM her, she IMs you back. Works. Simple. It works because the reference to your friend includes the URL of her world, so your world can forward the IM appropriately. And vice-versa.

Scenario #2: you jump to your friend’s world, you IM her and she IMs you back. No problem, that’s a local IM. Works. Simple.

Scenario #3: you go back to your world. Your friend IMs you. Not so simple! Your friend’s world needs to detect that you aren’t in her world anymore and needs to find out where you are. Your friend’s world fails once, but has a reference to you, including your world’s URL, so that’s what it uses to forward the IM.

Scenario #4: you HG-TP to a 3rd world. Your friend IMs you. More challenging! Now your world needs to find out where you are. Your User Agent service knows that, and forwards the IM to the world where you are.

Scenario #5: in that 3rd world, you IM your friend back. This is really challenging! — because that world knows absolutely nothing about your friend. No worries, it still works in this case. It works like this: that 3rd world knows about you, so it asks your world if it knows about the user you are trying to IM. Basically, all that world needs is a URL to deliver the IM. Prompted with that information request, your world first looks if the user is a co-resident of yours; if it is, it sends the URL of your world; if it’s not, as is the case in this scenario, your world checks if the requested user is a foreign friend of yours, which is the case here. Having found it, it sends the URL of your friend’s world to the 3rd world, so that the IM can finally be delivered.

Scenario #6: in that 3rd world you meet someone, and you start IM-ing with that person, who is not in your friend’s list. Then you go back to your world and you send an IM to that person in the window that is still open. Oops! — this won’t work! — at least for the time being. The reason it won’t work is because your world knows absolutely nothing about that person: it’s not a co-resident, it’s not visiting your world and it’s not in your friend’s list. Your world has no means to find the URL of that person’s world to send the IM to. Work in progress…

Note that throughout these IM interactions your location in the Hypergrid is known only to your user services; it is not disclosed to the worlds with whom you are IM-ing. Those other worlds simply forward the IMs to your world, and it’s your world’s responsibility to forward them to wherever you are. This is an important privacy-preserving design decision.

As I said, HG friends and IM is still work in progress. Use with caution, and be ready to go down to your DB and delete records in the friend’s table if things go awry. If all testing goes well, it will be in 0.7.2.

It has been a while since I made a D2 + Wifi releases. I had been waiting for OpenSim 0.7.1; the changes between 0.7.0.2 and 0.7.1 were such that I didn’t want to risk making a release in between. OpenSim 0.7.1 finally happened earlier this week, so here is the much awaited D2! Technically, it corresponds to OpenSim r15402, which is a couple of commits after the official release — I found a couple of bugs in the official release related to the Library, so I fixed them.

Things to pay attention to when you update your D2:

• This time after you Update, you need to run Configure again on your new installation, because of changes in configuration variables. In order for this process to be as smooth as possible, have your current config-include/MyWorld.ini open in one window while answering the questions for Configure in the new install. If you had made changes to your MyWorld.ini, simply add those changes to the new MyWorld.ini that is produced after running Configure.
• As usual, the Update tool will ask you if you want to preserve any changes you may have made to your Wifi pages. I always say “yes, I want to preserve my own.” By doing that, it’s guaranteed that the new Wifi installation will use the pages that I had before. However, this time there is a nice improvement in the splash.html page that shows some statistics at the bottom — see here for example. Not a problem. When Update copies over your old pages into the new install, it also places the new ones under WifiPages with extension .diva, for example, splash.html.diva. That way you can easily see the differences between the two and copy-and-paste the parts that matter. In my case, this time, I simply replaced the entire splash.html file with the new one; you may want to check your own changes.
• Marck added support for localization in Wifi (he’s German). Now you can have your Wifi installation in your favorite language. See here for how to do that (scroll down for the LOCALIZATION section).
• The Wifi administrator account has a teaser new link called “Console”, a new feature that we have been working on. It’s undocumented for the time being, because it’s not secure enough for me to recommend using it. It will come to life in a future release.
• Bad news: Teleports between OpenSim 0.7.0.x installations and OpenSim 0.7.1 installations are incompatible, because of a bug fix in libomv. You probably knew that if you had tried to HG TP to OSGrid.
• Good news: most large grids, like OSGrid, are already running 0.7.1, so you can finally join the crowd there.
• To test HG TPs, feel free to use my own world, http://ucigrid01.nacs.uci.edu:9001. It’s somewhere in the 5000’s and reachable from all D2’s if you haven’t changed your coordinates.

Have fun!

UPDATE (5/7): I realize that many people may not have noticed the subtle differences in HG addressing that happened between 0.6.9 (HG 1.0) and 0.7.0.x (HG 1.5), and then between 0.7.0.x and 0.7.1. Let me explain.

An HG address in the old HG 1.0 was simply the address of a particular simulator. HG 1.0 did not have the concept of “grid” in place, so even if a simulator was part of a grid, its HG address was the simulator’s address. For example, in the UCI grid you could reach individual simulators with addresses like ucigrid04.nacs.uci.edu:9007 (the Gateway 7000 region); note that ucigrid04 is the machine where that particular simulator runs, and 9007 is the HTTP port of that particular simulator.

With HG 1.5, the concept of “grid” has been put in place. A “grid” is a collection of simulators running under the same authority and that  share one single “Gatekeeper.” The Gatekeeper is central to HG 1.5 security and, in grids, it usually runs in the Robust server. The Gatekeeper is the entry point to every single incoming request like link region requests and agent transfer requests. In HG1.5 you cannot link directly to simulators anymore; you link to Gatekeepers wherever they live. For example the Gateway 7000 region in the UCI Grid is now accessed like this: http://ucigrid00.nacs.uci.edu:8002 “Gateway 7000″.  A few of things to notice:

• First, the “domain:port” syntax that was used before (and that is still accepted for the time being, but considered obsolete) has been replaced by a full-blown URL starting with “http://”. This is so that HG services can be implemented by regular web applications. For example, the Gatekeeper of ScienceSim lives in http://grid.sciencesim.com/grid/hypergrid.php — yes, a php script.
• Second, the URL is that of the Gatekeeper, not of the region itself. The region is referred to by name after the URL, and when the name has white spaces you need to use quotes. If you want to reach UCI Grid’s Gateway 7000 region, you can pull up the map in the viewer and type exactly what I just wrote in bold up there. Or if you want to use the link-region command on the sim console  you can type:
  $ link-region <local_x> <local_y> http://ucigrid00.nacs.uci.edu:8002 “Gateway 7000″
• Individual Gatekeepers may have different policies in place concerning incoming links, so a link request to specific regions inside a grid may very well fail; or it may be required. If you simply specify the URL of the Gatekeeper and omit the region name, the Gatekeeper is supposed to link you to a default region in that grid, if it is defined. If that’s not defined in the target grid, then the link request will fail.

For standalones like D2, there is no difference between “grid” and “simulator” — they’re one and the same. In standalones, the Gatekeeper runs inside the simulator. As a consequence, the HG address is that of the simulator itself. But  the points made above still apply, conceptually speaking.

UPDATE (5/8): well, foo! I found one wrong configuration variable in diva-r15402. I made a new release diva-r15402-b that corrects it. If you updated to diva-r15402, please update again, so that you get diva-r15402-b. No need to run Configure on this small update. Just run Update, switch to the new installation, run OpenSim from there and you’re done. (If you update from an older release, then you need to run Configure, as the instructions say).

Also, I found that it’s better to remove the old hyperlinks from the DB so that old formats don’t get in the way of your travels. It’s very simple. Login to your DB and issue this statement: DELETE FROM regions WHERE flags=524  (524 are hyperlinks)

It took a long time, but OpenSim 0.7 is finally here! It’s quite difficult to explain the value of refactorings and of  re-conceptualizations of the software, which is what happened to OpenSimulator between 0.6.x and 0.7. Those things tend to be intangible at first (“WTF? This does the same thing! And it has bugs that it didn’t have before!”, etc.); the ROI only shows up later.

As an example of the new interesting things that can now be done, this new release features an embedded Web application for handling user registrations. It’s called Wifi. The features of Wifi are quite simple:

• Account creation, optionally controlled by the administrator
• Configurable default avatars for new accounts
• Account updates by both users and administrator
• Account deletion by administrator
• Password recovery via email
• Simple user inventory management

These features are enough for the worlds I run; they’ll be enough for many people, I think. This is just the first release, I expect Wifi to improve as we go a long — especially if more people want to help… I welcome help on this!

See it in action in one of my test worlds. Feel free to create a dummy account, login to it, recover your password, etc. And login to the world afterward. You can also use it for testing your own installation of the diva distro by hypergriding to it. (Note: this world is just for demonstration of Wifi and of the latest diva distro release; it will be shut down in a couple of weeks)

Wifi has a few interesting properties that make it a good fit for small-to-medium OpenSimulator-based virtual worlds. First of all, it doesn’t require the installation of Apache or other Web servers; it’s all done within the OpenSimulator code base. If you’re like me, you will think of this as a major win. Not having to install an additional major component like Apache and PHP/Python/what-have-you means less administrative hassle, less load on the machine, and just plain joy. Web app! No Apache! Just OpenSimulator! Yey!

Second, using the OpenSimulator code base brings many technical benefits. This isn’t an immediate benefit, but it is a benefit in the long run. Unlike the Apache-based Web apps that interface with the DB directly, Wifi interfaces with the OpenSimulator services layer using the core code, and therefore it is isolated from future changes to the OpenSimulator DB schema. In short, and of interest to diva distro users, Wifi will probably never be in the situation that other apps of the same kind are, always on the verge of being discontinued due to the hassle of making it catch up with the OpenSim schema. In the long run, this is a very good thing.

Third, technically, Wifi is a set of components that can be loaded up by both standalone simulators and Robust servers. What this means is that the exact same thing can serve diva distro worlds as well as larger grids in one seamless environment, OpenSimulator. I haven’t packaged it up for grids, but I might consider doing it at some point. So if your world grows to a medium grid, Wifi can handle it with no hassle.

Enjoy!

And, of course, also enjoy the extra security in HG 1.5, but that is invisible.

Here are the relevant links:

• Latest diva distro
• How to set it up in 5 steps (6 for *ix)

If you already have a diva distro up & running, simply run Update.exe on it and follow the instructions. This time, it’s really important that you read the release notes.

One last important note: the sequence of upgrades has a ‘pivot’ release at r12751. In other words, r12751 is a mandatory stop from all older releases. From r12751 on, upgrade gets you the latest. This new release today is r13458.

The news came yesterday after lunch: major layoffs at Linden Lab, as much as 30% of their employees. Lindens who had been there from early on, respected engineers, all laid off. All of those who had, at some point, been involved in the idea of virtual world interoperability — gone. Then the new vision: Second Life on a browser, accessible to the masses via well-known social networks. Wow. This is what I call a 90-degree course adjustment.

Clearly, I know nothing about the internal situation at Linden Lab. Probably their VC money has dried off, maybe their revenue is not enough to pay so many people. Who knows what’s behind a 30% ‘rightsizing’… But the new vision is an indication that this is not just about balancing the budget sheet; it’s about redefining what Second Life is. LL’s CEO wants it to be more like FarmVille than like World of Warcraft. Too many people have commented on his vision, I’m not going to do it. He’s the head of the company, he should try to make his vision come to life.

What I want to talk about here is what this 90-degree course adjustment entails for OpenSim. I confess yesterday I had that familiar feeling of having reached the point of having to stand and lead. Not me, personally. But the OpenSim project, as a whole. The torch is on us. Let me explain.

In spite of being inherently a rebellious project, OpenSim has always existed in the shadow of Second Life. The rebellious nature was, indeed, just a confirmation of that dependency; the kind of relationship teenagers have with parents. A lot of people in the OpenSim community have been assuming that, sooner or later, the large Second Life virtual world would be trading avatars and money with OpenSim-based grids. The Open Grid Protocol (OGP) prototype, first unveiled in the summer of 2008, was a teaser; a promise of what could come next. Many OpenSim core developers have been deeply involved in efforts for virtual world interoperability, first in the Architecture Working Group, and, recently, in the VWRAP working group at the IEFT. Both of these working groups were led by Linden Lab engineers… who have been laid off.

In the wake of these layoffs, the idea that Second Life will be part of a larger web of virtual worlds is today more remote than ever. There’s no one left in Linden Lab to carry the interoperability torch, at least not of the kind we were thinking. It looks more likely that Second Life will be part of the Web itself, complete with the Web’s inability to deal with portable identity, and therefore eager to serve the largest pool of users on the Web, the 400 or so million Facebook users.

It’s always hard when the driver we were relying on suddenly makes a left turn from where we thought we were going. The question for OpenSim now is this: do we really want to follow Linden Lab on that left turn? Or do we take the torch and lead?

Personally, the last thing I want to do is to be involved in one more Facebook app. That’s the kind of interoperability project that I give my undergrad students on a 3-week time period. Not only it’s technically uninspired, but it’s missing the core of what I believe interoperable virtual worlds can bring to the Web itself: true, S2S portable identity. Not just “this is me” a-la OpenID, but “this is me, and I have a lot of baggage that I want to access while I’m visiting your site.” The vision of interoperable virtual environments is as exciting to me now as it was 3 years ago.

As usual, I don’t speak for the entire OpenSim project. And it’s probably too early to internalize what this course adjustment really means for OpenSim. But one thing is for sure: if there is going to be a web of virtual worlds, a decentralized S2S system of 3D environments that can seamlessly exchange user agents securely, that web will be made of OpenSim servers entirely, for the foreseeable future. Second Life is out of the picture.

In the past 2 weeks there has been a lot of plumbing in OpenSim, and things have improved considerably. First and foremost, we have identified and eliminated a memory leak that was causing OpenSim to use all memory over time, and eventually crash. Now OpenSim runs on much more reasonable memory footprint, and stays within that limit [for much longer].

Second, we have rewritten the grid service from scratch. The grid service is the part of OpenSim that manages region registration and lookup. All OpenSim installations have a grid service, even if they are standalones. In fact this distinction between standalone and “grid mode” is becoming fuzzier and fuzzier. So much so that it is now possible to have grids by stitching together standalone installations of OpenSim — without having to run any other servers!

Here are the instructions for how to do it with the diva distro. Instructions on how to do it with stock OpenSim can be found elsewhere.

Instructions for setting up a serverless grid with the Diva distro:

1 – If you want the second instance on the same machine, copy the entire folder of your diva distro into another folder on that machine. If you want the second instance on another machine, copy it to that other machine. Leave the original unchanged. In the copy (or copies), do the following steps.

2 – Edit Regions/RegionConfig.ini, and change the following fields for each region: (a) the name; (b) the RegionUUID; (c) the Location; and eventually (d) the InternalPort. For example, in my case my first region in my original instance is:

[Diva’s World 1]
RegionUUID = “9ca5b0b5-0b1e-47b7-ac30-7e0230152de0″
Location = “6178,3371”
InternalPort = 9000

In the copy on the same machine I have:

[Diva’s World 5]
RegionUUID = “854a1e57-95f2-42f9-b3c7-2488e0e0bd92″
Location = “6176,3371”
InternalPort = 9004

A couple of important notes:
* Please use truly random UUIDs. Here is a site that generates them for you: http://www.guidgenerator.com/.
* In my case I wanted to place the new 2×2 megaregion to the West of my original megaregion. Hence the coordinates 6176,3371 on the first region of the copy — the X coordinate is 2 to the left, the Y coordinate is the same.
* Megaregions need to be specified like what they are in that file: the first region is the SW corner, the second is the NW corner, the third is the SE corner and the fourth is the NE corner. Make sure you preserve this order when you make the changes in their coordinates.
* The InternalPorts must be unique for each region, on each machine. In my case, my second instance is on the same machine, so I had to change it. If your second instance in on another machine you can use the same InternalPort numbers of the original megaregion.

Do the necessary changes for the other three regions on that file.

3 – Edit config-include/MyWorld.ini. The only thing you need to change here, if at all, is in the [Network] section, the http_listener_port. If your second instance is on the same machine, you must change this port — for example 9001; this port needs to be unique per instance, per machine. If your second instance is on another machine, you can leave it as is. In any case, leave the server urls as they are for the original instance.

Et voila!

Login as usual. You will notice that the SW corner of your original instance now has neighbours to the left. In principle, you can cross back and forth, and teleport between regions as normal. In practice, crossing between instances with megaregions is still under heavy work and may produce surprising behaviour. For example, crossing from the original SW corner to the copy SE corner may end up being an auto-teleport, or you may get stuck in limbo, etc. Don’t be alarmed, this is “normal” behaviour for the time being, and these quirks will be fixed in future releases. For the time being,whenever you want to switch instances I recommend teleporting between SW corners of the different instances — that works pretty reliably.

I’m happy to announce Metaverse Ink’s “Diva Distribution”– a lean and mean OpenSim distribution targeting hypergrided standalones that comes preconfigured and that is very easy to keep up to date with newer releases of OpenSim. The Diva distribution is, literally, my personal preferences using the many configuration options of OpenSim. Here is what it consists of:

• MySQL as the DB backend. I used SQLite for a long time. SQLite is really easy to use, it requires no installation, it’s great for beginners. However, it’s clear that SQLite doesn’t scale well for long-term worlds. Moreover, using a MySQL server on the backend enables very interesting hybrid architectures that I will be talking about soon. The diva distribution requires some expertise for setting up MySQL, but it’s worth it. The instructions that come on the distribution make the MySQL setup for OpenSim really easy.
• ODE Physics. Easy choice here. Even the official OpenSim distribution is about to change to ODE as the default physics engine.
•  XEngine as the scripting engine, and lsl as the scripting language. XEngine is the most stable of the scripting engines we currently have, and lsl is the most widely used scripting language. All other language support is still experimental and fairly insecure.
• One Megaregion of 512×512 meters. That’s right! Megaregions just made it into OpenSim, thanks to Teravus, and they are working great. With megaregions there are no border crossings, everything is very smooth.
• MetaverseInk Search. If you want to make your world searcheable, mark your parcels “Show in search”, and they will be listed on MISearch’s engine. Others can teleport to your world from search results. Everyone can do this on their OpenSim by setting a certain configuration variable; the diva distribution has that preconfigured.

Over time we plan to add more goodies to the diva distribution, so stay tuned.

Additionally, the diva distribution contains two tools that make it really easy to configure and update your installation. They are:

• Configure: this is the tool you should run after unziping the first distribution. Be ready to tell it (a) the name you want to call your world; (b) the password for the MySql DB opensim account; (3) your externally visible IP address or domain name. Once you answer these questions, Configure will do all necessary configurations for you. You don’t need to edit any .ini files.

• Update: once you get the first installation set up, updating it is as easy as running the Update tool. Update asks no questions from you, it just does the right thing.

While the diva distribution comes preconfigured with my own preferences, it is possible to change these preferences to yours. For example, if you want more than a 512×512 megaregion, you can add more regions. The instructions for how to do that come in the documentation that is included in the distribution.

So where can you get it? Get it at http://github.com/diva/d2/downloads — the file to download is called diva-rNNNN.zip, which is listed at the bottom of that page.
Unzip it, read README, and follow the instructions from there.

Let me finish this post by reminding everyone that the current version of the Hypergrid, which we call HG1, is fairly insecure. Don’t take your favorite avatar to places that you don’t trust. HG2 is coming, but it’s not here yet.