(I’m still wrapping up the JISC part of SPAWS…normal blog service will resume shortly)
Here’s the original project proposal I submitted, including the project plan:
(I’m still wrapping up the JISC part of SPAWS…normal blog service will resume shortly)
Here’s the original project proposal I submitted, including the project plan:
The SPAWS project – or rather, the funded part – is coming to an end so its time to wrap up the project details.
In terms of impact, what have we achieved?
1. We answered the question we set: “is the LearningRegistry Node infrastructure suitable for sharing paradata between app stores?”. (tl;dr: yes).
2. We created a software library for it
3. We embedded the library into Edukapp, which means stores operated using it can switch on SPAWS and start syndicating paradata. As JISC, SURFNet and ITEC roll out their Edukapp-based stores between now and Q3 2013, we will start to see some real practical sharing benefits
5. We developed paradata recipes that have general applicability
7. We contributed to the evolution of the LR itself, in particular how it will handle duplicate/outdated records using TTL in future, enabling better scaling of this kind of paradata handling
Future impact now rests on the success of the stores using SPAWS.
The SPAWS project has produced:
All of which were in the original project plan.
Some things we didn’t do:
I had a good chat about this with my colleague Mark and I think I figured it out. PHP is the Pot Noodle of programming.
Let me explain. Pot Noodle is, frankly, a bit crap. On the other hand, the only thing it requires to make a meal is a kettle, which is pretty much the only appliance in the kitchen that is always guaranteed to be there, to work, and not be totally disgusting (especially if you’ve ever lived in a shared house). It also requires absolutely zero culinary skills. So as a student, Pot Noodle is a good standby. However, as a proper grown up I never really bother with them.
For me, PHP is a bit similar. All you need is a fairly normal web server, and it can do the job, and most people, including non-developers, can pick up enough skills in it fairly easily. (Looking at anything substantial built in PHP also looks a bit like a tangled mess of noodles wrapped around some unidentifiable chunks too.)
But its not something I particularly want to touch myself; which is fine, as if you throw a Pot Noodle out of the window of an IT dept there’s a reasonable chance it will land on someone who can do PHP.
So, if you need the job doing, if you can’t install a ton of software on the server, and it needs doing quickly… well there’s a web programming language that can fit.
Long live PHP!
Just don’t make me
eat use it.
Some of you may have noticed that I’ve just started a new role as Service Manager at OSS Watch and have been asking whether I’ve left CETIS. Well, the situation is that I’m being seconded to Oxford to work at OSS Watch for 80% of my time, for at least the next year (after that, its hard to say as it depends on funding).
So OSS Watch will be my main focus, but I haven’t stopped working on standards and specifications. You’ll also probably still meet me at some of the same kinds of events I’ve been attending in the past, just in a different role.
For some reason I never came across this before, but it seems there is something of a movement (complete with manifesto) promoting client-side-only web apps.
According to the Unhosted site:
By “unhosted web apps” we mean browser-based apps with no server-side backend. Unlike server-side or client-server apps, unhosted web apps leave users in control of their valuable user data and privacy, by default.
I’m a big fan of writing client-side apps using the web standards stack (JS, CSS, HTML) for many of the reasons stated on the site, including not least the fact I don’t need to provide an actual hosting platform – users can run the app locally, or access it from an app store (e.g. Edukapp) or a shared collaboration environment (e.g. Apache Rave).
Publishing a web application in this way means you can also make it available as open source software, allowing it to be forked and modified by users, and also potentially bringing in contributions and improvements from a wider community.
So, even when the intention is to create a web app that will work in a particular hosting runtime such as Apache Wookie, I always start with creating a working web app that will run in the browser without any server-side services. So, you can use the following apps directly in the browser or host them in Wookie:
(You can also run them locally in the browser – download the .wgt version, change the file extension to .zip and extract the contents)
I can then add tests for injected features provided by the runtime, and make use of them where available. For example, multi-user collaboration via the Wave API in Wookie, or inter-widget communication in Rave. Also even for things like storage, I can test whether widget.preferences is available, or default to localstorage. (Maybe in future I might want to give remotestorage a go, which provides more storage options for the user.)
I tend to opt wherever for possible for highly generic back-end services such as storage and communication, rather than creating a tightly-coupled set of services for each application. For some, this seems like an odd restriction, as you have so much freedom to do nice things on the server side (e.g. using NodeJS). However, it does make the apps much more portable and reusable, and avoids a lot of configuration hassles.
Even where it is necessary (at east for now) to use a server-based back-end, such as for collaboration or multi-user games, I created WaveNode as a generic backend in NodeJS so you don’t need to run a dedicated server-side component for each app, and the apps themselves will work anywhere there is a Wave Gadget API implementation (e.g. Apache Wookie, Apache Wave etc).
So I guess I’ve been in the Unhosted movement for a while now without realising it!
The funded part of SPAWS is coming to a close, so its time for me to write about some of the things I’ve learned so far from working on this project.
The major question this project posed was whether the LearningRegistry infrastructure could be used to effectively share paradata among a set of app stores. While we had some teething issues with the JLern test server, overall the infrastructure performed well, and the documentation was good enough to answer any questions I had.
There are some limitations we had to overcome; for example, there is no normalization of paradata, and so I had to code this into SPAWS itself. However I suspect its a useful enough capability to be eventually added to LR as an add-on module in the future.
The only aspect of the LR infrastructure we didn’t get to test was the federation/distribution aspect, where paradata is synchronised among a network of nodes rather than through a single node. Although from a SPAWS perspective I suspect it wouldn’t make any actual difference to the way it operates.
When I started I wasn’t sure whether we would be able to use existing JSON “recipes” for paradata, have to write our own, or whether to use CAM. In the end, I created some very simple recipes for the paradata we needed, published on Github. The basic ActivityStreams-inspired format used by LR seems good enough for the purpose, whereas CAM seemed more suited to more detailed learning analytics than the kind of basic usage information I needed to share, so I decided not to use CAM in SPAWS, at least not for the purpose of the initial project.
Something interesting that came up from our advisory group was the use of contextual information in the paradata – so links back to the page where the original comment was made, or the detail page for the widget, and to the public profile page of the user. This allows for stores (or other kinds of agents) to extract additional information where this may be useful, for example for analytics. This meant that we could keep the “core” paradata recipes very light and functionally-oriented (i.e., sticking to data that would actually be used in the UI or in core functions such as popularity ordering).
In the proposal I set out a business case for paradata sharing for “niche” app stores, somewhat speculatively at the time. Its good to see that this business case does seem to have some traction, and other commercial app stores have shown an interest as well as the initial set of educational app stores I focussed on for the project.
Timescales as always are something of a problem, with most of the stores I work with not going live (or even beta) for some months yet. However, there was enough of an opportunity to test out the SPAWS library in integration with the development sites of the stores to see it working OK.
There are potential privacy issues with sharing paradata that is personally generated, however I stepped around this by only sharing paradata that was either completely depersonalised (e.g. everyone’s total number of downloads/shares/likes for a widget) or completely public (e.g. published reviews). This is pretty superficial but functional as far as stores are concerned. However, by including a context URL to the original store page and user profile pages, its possible to harvest additional contextual information as needed; again this is all public rather than protected information. More detailed or personalised analytics are either not necessary for app stores at their current stage of development, or would be highly unlikely to be shared. So, the legal and privacy questions proved quite simple to resolve in this case.
I started out the code on Github from the very outset, not particularly with the idea of attracting contributions during the first six months (though that would be very welcome in future). The SPAWS library itself is published on OSSRH so is fairly simple to include in Java projects.
I also ended up contributing a fair number of pull requests to LRJavaLib, a generic LearningRegistry library that SPAWS builds on top of. There is currently quite a small Java community using LR so its especially important to use and contribute to existing libraries.
As it stands, SPAWS is quite a small codebase, with plenty of comments and testcases, so should be quite straightforward to sustain with minimal effort. Its also integrated into Edukapp, so will also be updated as needed for that project.
The LearningRegistry has a range of supported APIs; it provides simple HTTP “obtain” and “publish” services that use JSON, and it also supports OAI-PMH and SWORD. I’m not the biggest fan of OAI-PMH so I was quite relieved to find the simple obtain service was quite sufficient for my purposes; likewise for publishing the basic POST service was enough. I suspect that these protocols only really come into play when synchronising large volumes of metadata and paradata; for the SPAWS scenarios the amount of paradata will always be relatively small, certainly for each request, and not worth the extra hassle of having to maintain state on the client side.
In the end, I opted for using a cacheing strategy for “external” paradata rather than to actually synchronise it internally with the store’s own data, avoiding any potential conflicts. Perhaps on a larger scale it would make sense to create an actual local repository for external paradata and synch it using OAI-PMH, but that would seem to be quite a way off – the next logical scaling step for Edukapp, for example, would simply be to move from memory cache to disk cache, and increase the cache size.
Given that I didn’t need to use OAI-PMH or SWORD, the standards of interest were principally the widget specs themselves – W3C and OpenSocial. The stores already provide an abstraction of the common metadata for the two specs so it wasn’t really necessary for SPAWS to rely on any particular features of the specifications apart from one rather critical one – there is no “identifier” for an OpenSocial gadget.
This is quite an issue to overcome; I opted for relying on the URL for now while punting the issue to the OpenSocial spec community to see if it could be addressed there.
Historically, each OpenSocial gadget was hosted on its own site rather than packaged and hosted elsewhere; however over time there have been gadgets being packaged up and rehosted and so this would become more of a requirement – perhaps even using the W3C format to package and transport OpenSocial gadgets.
I can also see now that what I’ve worked on here for app/widget stores would also work fine for pretty much any kind of site – so product catalogues or more traditional repositories could also reuse it pretty much as-is, including the recipes. I was originally expecting it to be similar but not directly reusable; this is something worth pursuing .
Given what I’ve learned, would I do anything differently? I don’t think so, I think this project has answered the questions we posed initially, so its more now a question of what to do next.
However, what I most want to do now is get the code into production in the stores that are being launched and take it from there.
I’ve finally gotten around to creating a quick demo of one of the use cases I wrote for SPAWS – sharing a comment.
So, here are the steps in the use case:
Seems easy enough – lets go!
Here’s a widget I’m going to review in my local Edukapp:
Well, I don’t really need to do this, as I can play with a live preview on the site, but lets take it as read I’ve had a go with the widget, and have formed an opinion.
So, I create an account and log into Edukapp, and submit a review; here it is:
On ITEC, lets find the Weather widget; luckily its pretty obvious on the “featured” page:
Yep, there it is!
Note that, because its an “external” review, ITEC hasn’t provided a link for the user who submitted it. SPAWS does send a URL with the “Agent” paradata that links to the user profile back on the original site, but its up to stores whether they want to use that, or to just style external reviews without any links.
Note also that you can set up a similar kind of demo using two different instances of Edukapp (just make sure in store.properties you give each one a different value for “spaws.submitter.name” or they won’t show up in the other store as it filters out any paradata it thinks it submitted itself).
This year for the first time we had a Google Summer of Code project in Apache Wookie - Pushpalanka Jayawardhana took on the challenge of implementing the W3C XML Digital Signatures for Widgets specification, which describes how to sign and verify a W3C Widget.
Of course, actually implementing the specification in Wookie required a lot more than just the signing and verification routines, but a complete workflow, including how to sign a widget as an author, how to configure the verification options for the Wookie server application, and how to establish whether to trust a widget that contains a valid digital signature. So rather than just following a published specification, the project also involved a lot of design challenges.
The end result is a suite of components and enhancements for Wookie: Pushpalanka created a standalone application to help authors sign their widgets, added signature verification capabilities to the Wookie W3C Widget parser library, and implemented a configurable verification workflow in the server code itself. You can read a detailed description of the design and implementation over on Pushpalanka’s blog.
As a result, a Wookie server administrator can now choose whether to check whether widgets added to Wookie have digital signatures, whether to reject widgets that have invalid signatures or just log a warning, and whether to require widgets to be signed by a trusted party whose certificate is included in the server keystore.
Thanks Pushpalanka for adding this great set of features to Wookie, and thank you Google for sponsoring this effort through Google Summer of Code!
Whether its live streaming of status updates, chatting live with friends, or playing games, there has been a significant trend for adding more real-time capabilities to the web, something that has been noticeable since the introduction of instant messaging and microblogging. (ReadWriteWeb has an article from 2009 heralding the realtime web… and a lot of the information there is still relevant)
At one time these would require dedicated, proprietary backend solutions using Flash (and its associated RTSP protocol) or plugins, however there has been a strong push towards providing more web-native capabilities. At low level, the development of Web Sockets, and supporting technologies such as Socket.io and Node.js, has driven experimentation with real-time services for the web. However, this is still fundamentally a client-server technology, albeit with a stream-oriented twist; however there have been developments lately that offer both peer-to-peer and cloud-driven realtime services.
W3C has for some time now been working on standardising Web Real-Time Communications through a dedicated WEBRTC working group. However, its a sign of just how hot this topic is becoming that already Microsoft has submitted a competing RTC specification to the group.
Venture capital interest is also evident with Realtime getting $100m funding. Realtime provides a cloud-based event bus for real-time updates, effectively providing a websocket farm for application developers. Its not the first entrant in this area; for example Pusher, BeaconPush, Echo and PubNub have been around for a while, and gained a reasonable amount of adoption. However it does indicate that technology investors are increasingly confident that real-time is the future of the web.
What this means for app developers and providers of services is a wealth – and burden – of choices for adding a real-time user experience. However, the common denominators are the core web standards such as websockets, which most other realtime technologies build upon. The cloud-based solutions linked above, for example, largely provide a wrapper around a web sockets interface, connecting to their hubs for routing data to each client.
WebRTC, however, provides a rather different set of possibilities: given its ability to support peer-to-peer streaming, it lends itself very well to add-hoc collaboration such as whiteboards and video chat, and could well replace many of the – often expensive – existing solutions widely used in enterprise and education alike.
Looking ahead, its hard to say whether there will be a consolidation of WebRTC around one of the competing protocols, or a face-off between the two camps. Likewise, there may be consolidation among the cloud-based providers, or even more new entrants as investors take notice. One thing is for certain though, and thats that the web is going real time, real fast.