DERI Pipes: Open Source, Extendable, Embeddable Web Data Mashups

 Inspired by Yahoo's Pipes, DERI Pipes is an engine and graphical environment for general Web Data transformations and Mashup.

• Supports RDF, XML, Microformats, JSON and binary streams.
• Use it as a "Web Pipe" or embedded in your applications
• Works as a mashup command Line tool
• Supports SPARQL, XQUERY, Several scripting languages.
• Extend it as needed

DERI Pipes, in general, produce as an output streams of data (e.g. XML, RDF,JSON) that can be used by applications. However, when invoked by a normal browser, they provide a end user GUI for the user to enter parameter values and browse the results (see the examples below) .

 

Pipe examples in XML (see specification )

Pipes are written in a simple XML language to specify the operators in the pipe and the connections between them. E.G. a mixing data from 2 sources is as simple as this:

<pipe>
   <simplemix>
     <source><fetch><location>http://www.w3.org/People/Berners-Lee/card</source></fetch></location>
     <source><fetch><location>http://g1o.net/foaf.rdf</source></fetch></location>
   </simplemix>
</pipe>

The output of this pipe is calculated live when you fetch http://pipes.deri.org:8080/pipes/Pipes/?id={pipeid}.

The Construct block allows RDF Transformation by the means of SPARQL Construct queries. Moreover, Pipes operators can be then nested as needed. In the following example Tim Berners-Lee's DBLP record is fetched and transformed to use his real URI:

<pipe>
 <construct>
  <source><fetch><location>http://dblp.l3s.de/d2r/resource/authors/Tim_Berners-Lee</source></fetch></location> 
  <query>
       <![CDATA[
          CONSTRUCT {<http://www.w3.org/People/Berners-Lee/card#i> ?p ?o. 
                     ?s2 ?p2 <http://www.w3.org/People/Berners-Lee/card#i>} where 
         {{<http://dblp.l3s.de/d2r/resource/authors/Tim_Berners-Lee> ?p ?o} UNION  
          {?s2 ?p2 <http://dblp.l3s.de/d2r/resource/authors/Tim_Berners-Lee>}}
	 ]]>
   </query>
 </construct>
</pipe>

Pipes are easily defined using a graphical DERI Pipes Editor. In this environment pipes are created by drag-and-drop operators onto a panel then connecting operators to create pipes. The following figure is a screen shot of DERI Pipes Editor of the pipe mentioned above (click on the picture to see the screencast).

The output of this pipe is available at http://pipes.deri.org:8080/pipes/pipes/?id=TBLonTheSW.

A list of pipes is available in the Pipe editor, each with its own stable invocation URL. To prevent unwanted deletion or editing, pipes can be password protected.

People:

• Danh Le Phuoc (DERI Galway)
• Christian Morbidoni (Semedia, Universita' Politecnica delle Marche, Ancona)
• Axel Polleres (DERI Galway)
• Matthias Samwald
• Robert Fuller (DERI Galway)
• Giovanni Tummarello (DERI Galway)