Semantic Web roadmap

Semantic Web Road map
A road map for the future, an architectural plan untested by anythingexcept thought experiments.
This was written as part of a requested road map for future Web design,from a level of 20,000ft. It was spun off from an Architectural overview foran area which required more elaboration than that overview could afford.
Necessarily, from 20,000 feet, large things seem to get a small mention.It is architecture, then, in the sense of how things hopefully will fittogether. So we should recognize that while it might be slowly changing, thisis also a living document.
This document is a plan for achieving a set of connected applications fordata on the Web in such a way as to form a consistent logical web of data(semantic web).
 
The Web was designed as an information space, with the goal that it shouldbe useful not only for human-human communication, but also that machineswould be able to participate and help. One of the major obstacles to this hasbeen the fact that most information on the Web is designed for humanconsumption, and even if it was derived from a database with well definedmeanings (in at least some terms) for its columns, that the structure of thedata is not evident to a robot browsing the web. Leaving aside the artificialintelligence problem of training machines to behave like people, the SemanticWeb approach instead develops languages for expressing information in amachine processable form.
This document gives a road map - a sequence for the incrementalintroduction of technology to take us, step by step, from the Web of today toa Web in which machine reasoning will be ubiquitous and devastatinglypowerful.
It follows the note on thearchitecture ofthe Web, which defines existing design decisions and principles for what hasbeen accomplished to date.
 
The Semantic Web is a web of data, in some ways like a global database.The rationale for creating such an infrastructure is given elsewhere [Webfuture talks &c] here I only outline the architecture as I see it.
When looking at a possible formulation of a universal Web of semanticassertions, the principle of minimalist design requires that it be based on acommon model of great generality. Only when the common model is general canany prospective application be mapped onto the model. The general model isthe Resource Description Framework.
See theRDF Model and SyntaxSpecification
Being general, this is very simple. Being simple there is nothing much youcan do with the model itself without layering many things on top. The basicmodel contains just the concept of an assertion, and the concept ofquotation - making assertions about assertions. This is introducedbecause (a) it will be needed later anyway and (b) most of the initial RDFapplications are for data about data ("metadata") in which assertions aboutassertions are basic, even before logic. (Because for the target applicationsof RDF, assertions are part of a description of some resource, that resourceis often an implicit parameter and the assertion is known as aproperty of a resource).
As far as mathematics goes, the language at this point has no negation orimplication, and is therefore very limited. Given a set of facts, it is easyto say whether a proof exists or not for any given question, because neitherthe facts nor the questions can have enough power to make the problemintractable.
Applications at this level are very numerous. Most of theapplications for the representation ofmetadata can be handled by RDF at this level. Examples include card indexinformation (the Dublin Core), Privacy information (P3P), associations ofstyle sheets with documents, intellectual property rights labeling and PICSlabels. We are talking about the representation of data here, which istypically simple: not languages for expressing queries or inference rules.
RDF documents at this level do not have great power, and sometimes it isless than evident why one should bother to map an application in RDF. Theanswer is that we expect this data, while limited and simple within anapplication, to be combined, later, with data from other applications into aWeb. Applications which run over the whole web must be able to use a commonframework for combining information from all these applications. For example,access control logic may use a combination of privacy and group membershipand data type information to actually allow or deny access. Queries may laterallow powerful logical expressions referring to data from domains in which,individually, the data representation language is not very expressive. Thepurpose of this document is partly to show the plan by which this mighthappen.
The basic model of the RDF allows us to do a lot on the blackboard, butdoes not give us many tools. It gives us a model of assertions and quotationson which we can map the data in any new format.
We next need a schema layer to declare the existence of new property. Weneed at the same time to say a little more about it. We want to be able toconstrain the way it used. Typically we want to constrain the types of objectit can apply to. These meta-assertions make it possible to do rudimentarychecks on a document. Much as in SGML the "DTD" allows one to check whetherelements have been used in appropriate positions, so in RDF a schema willallow us to check that, for example, a driver‘s license has the name of aperson, and not a model of car, as its "name".
It is not clear to me exactly what primitives have to be introduced, andwhether much useful language can be defined at this level without alsodefining the next level. There is currently aRDF Schema working group inthis area. The schema language typically makes simple assertions aboutpermitted combinations. If the SGML DTD is used as a model, the schema can bein a language of very limited power. The constraints expressed in the schemalanguage are easily expanded into a more powerful logical layer expressions(the next layer), but one chose at this point, in order to limit the power,not to do that. For example: one can say in a schema that a property foo isunique. Expanded, that is that for any x, if y is the foo of x, and z is thefoo of x, then y equals z. This uses logical expressions which are notavailable at this level, but that is OK so long as the schema language is,for the moment, going to be handled by specialized schema engines only, notby a general reasoning engine.
When we do this sort of thing with a language - and I think it will bevery common - we must be careful that the language is still well definedlogically. Later on, we may want to make inferences which can only be made byunderstanding the semantics of the schema language in logical terms, andcombining it with other logical information.
 
A requirement of namespaces work forevolvability is that one must, with knowledge ofcommon RDF at some level, be able to follow rules for converting a documentin one RDF schema into another one (which presumably one has an innateunderstanding of how to process).
By the principle of least power, this language can in fact be made to haveimplication (inference rules) without having negation. (This might seem afine point to make, when in fact one can easily write a rule which definesinference from a statement A of another statement B which actually happens tobe false, even though the language has no way of actually stating "False".However, still formally the language does not have the power needed to writea paradox, which comforts some people. In the following, though, as thelanguage gets more expressive, we rely not on an inherent ability to makeparadoxical statements, but on applications specifically limiting theexpressive power of particular documents. Schemas provide a convenient placeto describe those restrictions.)
 
A simple example of the application of this layer is when twodatabases, constructed independently and then put on the web, are linked bysemantic links which allow queries on one to converted into queries onanother. Here, someone noticed that "where" in the friends table and"zip" in a places table mean the same thing. Someone else documentedthat "zip" in the places table meant the same things as "zip" in theemployees table, and so on as shown by arrows. Given thisinformation, a search for any employee called Fred with zip 02139 can bewidened from employees to in include friends. All that isneeded some RDF "equivalent" property.
The next layer, then is the logical layer. We need ways of writing logicinto documents to allow such things as, for example, rules the deduction ofone type of document from a document of another type; the checking of adocument against a set of rules of self-consistency; and the resolution of aquery by conversion from terms unknown into terms known. Given that we havequotation in the language already, the next layer is predicate logic (not,and, etc) and the next layer quantification (for all x, y(x)).
The applications of RDF at this level are basically limited only by theimagination. A simple example of the application of this layer is when twodatabases, constructed independently and then put on the web, are linked bysemantic links which allow queries on one to converted into queries onanother. Many things which may have seemed to have needed a new languagebecome suddenly simply a question of writing down the right RDF. Once youhave a language which has the great power of predicate calculus withquotation, then when defining a new language for a specific application, twothings are required:
One must settle on the (limited) power of the reasoning engine which the receiver must have, and define a subset of full RDF which will be expected to be understood;
One will probably want to define some abbreviated functions to efficiently transmit expressions within the set of documents within the constrained language.
See also, if unconvinced:
What the Semantic Web is not - answering some FAQs
The metro map below shows a key loop in the semantic web. The Web part, onthe left, shows how a URI is, using HTTP, turned into a representation of adocument as a string of bits with some MIME type. It is then parsed into XMLand then into RDF, to produce an RDF graph or, at the logic level, a logicalformula. On the right hand side, the Semantic part, shows how the RDF graphcontains a reference to the URI. It is the trust from the key, combined withthe meaning of the statements contained in the document, which may cause aSemantic Web engine to dereference another URI.

The RDF model does not say anything about the form of reasoning engine,and it is obviously an open question, as there is no definitively perfectalgorithm for answering questions - or, basically, finding proofs. At thisstage in the development of the Semantic Web, though, we do not tackle thatproblem. Most applications construction of a proof is done according to somefairly constrained rules, and all that the other party has to do is validatea general proof. This is trivial.
For example, when someone is granted access to a web site, they can begiven a document which explains to the web server why they should haveaccess. The proof will be a chain [well, DAG] of assertions and reasoningrules with pointers to all the supporting material.
The same will be true of transactions involving privacy, and most ofelectronic commerce. The documents sent across the net will be written in acomplete language. However, they will be constrained so that, if queries, theresults will be computable, and in most cases they will be proofs. The HTTP"GET" will contain a proof that the client has a right to the response. theresponse will be a proof that the response is in deed what was asked for.
RDF at the logical level already has the power to express inference rules.For example, you should be able to say such things as "If the zipcode of theorganization of x is y then the work-zipcode of x is y". As noted above, justscattering the Web with such remarks will in the end be very interesting, butin the short term won‘t produce repeatable results unless we restrict theexpressiveness of documents to solve particular application problems.
Two fundamental functions we require RDF engines to be able to do are
for a version n implementation to be able to read enough RDF schema to be able to deduce how to read a version n+1 document;
for a type A application developed quite independently of a type B application which has the same or similar function to be able to read and process enough schema information to be able to process data from the type B application.
(Seeevolvability article)
The RDF logic level is sufficient to be usable as a language for makinginference rules. Note it does not address the heuristics of any particularreasoning engine, which which is an open field made all the more open andfruitful by the Semantic Web. In other words, RDF will allow you to writerules but won‘t tell anyone at this stage in which order to apply them.
Where for example a library of congress schema talks of an "author", and aBritish Library talks of a "creator", a small bit of RDF would be able to saythat for any person x and any resource y, if x is the (LoC) author of y, thenx is the (BL) creator of y. This is the sort of rule which solves theevolvability problems. Where would a processor find it? In the case of aprogram which finds a version 2 document and wants to find the rules toconvert it into a version 1 document, then the version 2 schema wouldnaturally contain or point to the rules. In the case of retrospectivedocumentation of the relationship between two independently invented schemas,then of course pointers to the rules could be added to either schema, but ifthat is not (socially) practical, then we have another example of the theannotation problem. This can be solved by third party indexes which can besearched for connections between two schemata. In practice of course searchengines provide this function very effectively - you would just have to ask asearch engine for all references to one schema and check the results forrules which like the two.
One is a query language. A query can be thought of as an assertion aboutthe result to be returned. Fundamentally, RDF at the logical level issufficient to represent this in any case. However, in practice a query enginehas specific algorithms and indexes available with which to work, and cantherefore answer specific sorts of query.
It may of course in practice to develop a vocabulary which helps in eitherof two ways:
 
It allows common powerful query types to be expressed succinctly with fewer pages of mathematics, or
It allows certain constrained queries to be expressed, which are interesting because they have certain computability properties.
SQL is an example of a language which does both.
It is clearly important that the query language be defined in terms of RDFlogic. For example, to query a server for the author of a resource, one wouldask for an assertion of the form "x is the author of p1" for some x. To askfor a definitive list of all authors, one would ask for a set of authors suchthat any author was in the set and everyone in the set was an author. And soon.
In practice, the diversity of algorithms in search engines on the web, andproof-finding algorithms in pre-web logical systems suggests that there willin a semantic web be many forms of agent able to provide answers to differentforms of query.
One useful step the specification of specific query engines for forexample searches to a finite level of depth in a specified subset of the Web(such as a web site). Of course there could be several alternatives fordifferent occasions.
Another metastep is the specification of a query engine descriptionlanguage -- basically a specification of the sort of query the engine canreturn in a general way. This would open the door to agents chaining togethersearches and inference across many intermediate engines.
Public key cryptography is a remarkable technology which completelychanges what is possible. While one can add a digital signature block asdecoration on an existing document, attempts to add the logic of trust asicing on the cake of a reasoning system have to date been restricted tosystems limited in their generality. For reasoning to be able to take trustinto account, the common logical model requires extension to include the keyswith which assertions have been signed.
Like all logic, the basis of this, may not seem appealing at first untilone has seen what can be built on top. This basis is the introduction of keysas first class objects (where the URI can be the literal value of a publickey), and a the introduction of general reasoning about assertionsattributable to keys.
In an implementation, this means that reasoning engine will have to betied to the signature verification system . Documents will be parsed not justinto trees of assertions, but into into trees of assertions about who hassigned what assertions. Proof validation will, for inference rules, check thelogic, but for assertions that a document has been signed, check thesignature.
The result will be a system which can express and reason aboutrelationships across the whole range of public-key based security and trustsystems.
Digital signature becomes interesting when RDF is developed to the levelthat a proof language exists. However, it can be developed in parallel withRDF for the most part.
In the W3C, input to the digital signature work comes from manydirections, including experience with DSig1.0 signed "pics" labels, andvarious submissions for digitally signed documents.
Given a worldwide semantic web of assertions, the search engine technologycurrently (1998) applied to HTML pages will presumably translate directlyinto indexes not of words, but of RDF objects. This itself will allow muchmore efficient searching of the Web as though it were one giant database,rather than one giant book.
The Version A to Version B translation requirement has now been met, andso when two databases exist as for example large arrays of (probably virtual)RDF files, then even though the initial schemas may not have been the same, aretrospective documentation of their equivalence would allow a search engineto satisfy queries by searching across both databases.
While search engines which index HTML pages find many answers to searchesand cover a huge part of the Web, then return many inappropriate answers.There is no notion of "correctness" to such searches. By contrast, logicalengines have typically been able to restrict their output to that which isprovably correct answer, but have suffered from the inability to rummagethrough the mass of intertwined data to construct valid answers. Thecombinatorial explosion of possibilities to be traced has been quiteintractable.
However, the scale upon which search engines have been successful mayforce us to reexamine our assumptions here. If an engine of the futurecombines a reasoning engine with a search engine, it may be able to get thebest of both worlds, and actually be able to construct proofs in a certainnumber of cases of very real impact. It will be able to reach out to indexeswhich contain very complete lists of all occurrences of a given term, andthen use logic to weed out all but those which can be of use in solving thegiven problem.
So while nothing will make the combinatorial explosion go away, many reallife problems can be solved using just a few (say two) steps of inference outon the wild web, the rest of the reasoning being in a realm in which proofsare give, or there are constrains and well understood computable algorithms.I also expect a string commercial incentive to develop engines and algorithmswhich will efficiently tackle specific types of problem. This may involvemaking caches of intermediate results much analogous to the search engines‘indexes of today.
Though there will still not be a machine which can guarantee to answerarbitrary questions, the power to answer real questions which are the stuffof our daily lives and especially of commerce may be quite remarkable.
 
In this series:
What the Semantic Web is not - answering some FAQs of the unconvinced.
Evolvability: properties of the language for evolution of the technology
Web Architecture from 50,000 feet
The CYCRepresentation Language
KnowledgeInterchange Format (KIF)