Technology Ecosystems

time

I feel a bit embarrassed having only secured one single blog post in 2013 . My mentor Martin Weller would be ashamed of me. Interestingly though, according to my annual feedback, my blog received almost as many visits in 2013 as 2012 (around 3000 visitors). Is that a reflection that the content is becoming more valuable as time passes?

Reflecting on 2013, for me it’s been the year of turning aspirations into products. For the iSpot project for example I created what I called a “Technical Roadmap”, which is really a grand way of saying that we had so much to deliver from a total of four different funders, we also been involved in BBC TV series (The Great British Year) and in the OU’s first Futurelearn MOOC on Ecosystems. (Which I took part but sadly became a drop-out!)

As a consequence we needed to ramp up the technical management of the project for what was an extremely challenging year and the Technical Roadmap helped us to keep our sanity (most of the time). Richard Greenwood has created a project blog about the main technical work during 2013.

Here are a some of my highlights:

globe1. Internationalisation/Community (the link takes you to the UK and Ireland community)- This is by far the biggest technical feat of the year for iSpot. The system now supports numerous communities organised according to geographical or taxonomic criteria. Richard Greenwood worked very hard on the functionality, which uses polygon mapping to calculate areas (and use multiple polygons so a region such as the UK, or Eastern Europe can be mapped out). The difficulty was providing communities without destroying the taxonomy (species dictionaries) as these sometimes span many areas. With the UK is was simple but now there are multiple dictionaries (one for Global iSpot) that need to be used in the correct places. Richard therefore couples the taxonomies to the observations locations, but decoupled it from the community (polygon) model, thus allowing freedom to create communities without having to use a dictionary that wasn’t relevant to their locale. The technology used is MariaDB and Open Street Map for creating polygons (and Google maps for displaying them). Richard also implemented Geo-IP to direct people to the correct community be default and the system will also allow people to move to different communities. Communities don’t have to be countries (we now have a budding Chilean community on iSpot for example ). Communities have their own News items and maps which are centred on their geographical region, and observations relevant to that community. Communities don’t just have to be geographical, they can also be around organisations or species or in fact anything that can be filtered against within iSpot, this makes the feature potentially very powerful.

species surfer2. A species surfer – The species surfer (or ID tool as it was originally called) allows anyone on iSpot to browse the species dictionary (taxonomy) using images to represent the main categories and sub-categories. Within a sub-category people can look at the variety of types to track down ones that are similar to their own observations. We know from talking to users that this is something they’ve been interested in having. Many people use Google and other sites to try to find out more about their observations and we thought that since iSpot has over 250,000 observations, the majority of which have been accurately identified, we should use that feature and draw it to people’s attention. It also acts as a learning tool and we hope it will be useful for field studies and research, from novices through to experts. This has only just been released so we still have further work to do to improve it but we want to get feedback from users since we know that there is still more work to do on this. The iSpot team have  provided  help information to guide people in how to use it correctly.

quiz13. Intelligent quiz – The existing crowdsourced identification model within iSpot, rewarding improvement in ability to identify observations, provides some of evidence that people are learning and improving their understanding of nature through iSpot, however it isn’t full-proof. For example a person may gain reputation through identifying very common species and without expanding their knowledge of other species. We therefore require empirical evidence of improvement in people’s ability to identify a greater variety of observations as their reputation improves; the iSpot intelligent quiz is designed to test this knowledge. The quiz was launched in July 2013, since then around 350 people per week have taken one or more quizzes, so an average of around 50 people per day. The quiz is tailored to the level and subject area that people request when they start a new quiz on iSpot. The reputation level that iSpot provides is a good indicator of the level that people should take but there is no restriction on the level so, for example, a level five expert could take  a level 1 quiz and vice versa. The data from the weekly logs shows however the people are averaging about 7 out of ten for quizzes across the skills levels which suggests that people are naturally finding a level which challenges them.

The quiz has a number of different types of question that test a range of knowledge within a specific domain, some questions are multiple choice and others are about entering the correct name or type of observation, some examples are shown below:-

quiz2

The quiz is largely image-based and relies on people correctly identifying observations. The quiz is open to both visitors to the website who have not yet registered, and also to registered users. Registered users have the benefit of being able to look back at previous quizzes they have taken to compare results. As part of the intelligence the quiz tries to select images which have been agreements and ones which are non-contentious, for example it will attempt to filter out hybrid types. In the example below people can use the button in the right hand corner of the image to expand it and see additional detail.

quiz3

Certain questions prompt people to enter correct names associated with an image, they are based on the names given within the species dictionary on iSpot. The system will look up the dictionary and offer suggestions for entries that match, or which are very similar to, the name entered by the user.

quiz4

We collect overview information about the quizzes on a weekly basis, including information about preferred groups, as you can see from the chart below birds consistently prove to be the most popular category for people taking the quiz.

quiz5

quiz6

The weekly statistics show us that the percentage of visitors who take quizzes compared to registered users varies from week to week.

For example during w/c 16th September 2013 about three quarters of people taking the quiz are registered users as indicated in the following diagram.

Interestingly during the previous week the ratio was more like 60/40 in favour of registered users so this seems to be indicating that as time passes the quiz may be becoming more popular with registered users however this will require further data analysis.

quiz7

Each quiz has up to ten questions so the table below shows that during the previous week there is an 80.7% completion rate.

The completion rate for the previous week was 89.1% and completion rates seem to fall consistently within 80%-89% percent range.

quiz8We are tracking the average scores of people who take the quiz and the results show us that there is only a very slight variation in score between people who class themselves as novice and take a level 1 quiz and people who class themselves as expert and take the level 5 quiz.

There is a slight decrease from 7.5 to 6.5 going from level 2 to level 3 and beyond however it is worth bearing in mind that the quiz provides novice users with up to three “lifelines” to use to help them (a lifeline is typically where two of the four choices are removed to make it simpler for people to find the correct remaining answer).

We have yet to analyse the raw data coming from the quizzes and because the service is relatively new we need more time before we can start to get useful trend data to help us demonstrate that people are increasing in their knowledge of nature through using iSpot.

In particular we need to understand the relationship between the amount of time people have been using iSpot and the level of knowledge they have attained. The data already indicates that people who use iSpot are gaining knowledge about nature and over the next few months we will be conducting further data analysis to understand exactly how this is being achieved.

These are just a selection of some of the new features in iSpot (I have at least 24 more to share with you!). I am very interested in how these systems evolve over time and the nature of the co-evolution of the technology and the people using that technology.

The Facebook we see today is very different from the first iteration of Facebook.

People are generally much more technology aware, and use technologies frequently for “selfies” and to share with others in a connected way. Systems must therefore evolve to support the changing perceptions of users to technology and iSpot can naturally support learning using images and photographs that people nowadays naturally want to share.

I’ve summarised some of the latest iSpot features that explain this co-evolution process in a presentation that I gave in December. We “technocrats” rely heavily on the community, and the subject experts to help us create services that are useful and provide mechanisms of learning and improvement.

I will be continuing  over the coming months to give examples of the richness of the  systems that we’re working on the Institute of Educational Technology. Working in partnership with the Science Faculty and Open Media Unit and the 36,000 users of iSpot.

Advertisements

Personae gratae

A group of us who are involved in developing the future learning system plans for the Open University are using a range of techniques taken from “User Centred Design” and User Experience (UX) to help us create the future systems for the OU and also to explain the complexity of the systems developments to senior management in a way that is easily understood and powerful. I wanted to share some of these techniques that we’re using without going into any of the detail which may be business sensitive.

man with hammer image

First of all what we’re doing is using a combination of Personas (some people suggest personae as the plural but I’ll use personas to describe these) and scenarios. There are many websites and blog posts going back years which talk about the power of personas and scenarios to design and development. JISC have used it within their design workshops and they’re used in different ways by different groups, for example here’s a post on “Web Design from scratch” by Ben Hunt which describes their use in design.

We’re using these in a slightly different way than for design but rather to describe areas of functionality to be developed to meet particular needs. In the persona development we adopted a range of persona’s that were created by the Online Communications team to describe target users for OU websites.

Here’s an example snippet of one of the persona’s to help explain them…

Jason
Gamer
Age/personal:  18, lives in Glenrothes with his Mum
Job:  Works in Dixon’s part-time
Education:  Highers
Studying aim: Degree in Computing/IT
Online likes:  Interaction, multimedia,
customisation and iPhone apps
Web games, chats, texts; surfs fast, but without
direction

Jason?

We use a set of personas to describe a range of target users and they test the system through a typical use case. We also have some high level scenarios to describe the depth of a particular system in supporting users from end-to-end. Scenarios in our case describe the environmental elements not possible easily through personas, so our scenarios are focused on direction setting and understanding where the OU should be going to meet the demands of new learners. for example we have scenarios based around informal learning becoming prevalent and another scenario around the need for key skills.

Personas are powerful because they:-

  • Allow systems to be developed to meet specific user types
  • Afford consistency of development across different systems
  • Are a useful tool for describing how people will use the services
  • Are useful for testing and benchmarking services against requirement, i.e. are useful for usability and accessibility testing.
Scenarios are powerful to us because they:-
  • Describe the full end-to-end functionality of a system
  • Take socio-economic and other environmental factors into account
  • Set direction of development
  • Describe the strategic value and business benefits

We are using these to map through to a set of “Roadmaps” which describe how we intend to deliver the changes. The roadmaps, programmes and projects within it are along the lines of the JISC P3 model which itself is a variant of PRINCE 2 methodology and therefore well established. The creative bit is how we’re describing this through the combination of personas and scenarios. We have been through this process once before with a programme called RAP (Roadmap Acceleration Programme) where we used a world cafe approach to gathering requirements (see my previous post on Future Learning Systems ). We used the user testing sessions to “validate” the personas against real people to ensure that they’re accurate and complete and the testing informs the system development, this was particularly useful to establish what works in the less clearly defined areas of the roadmap such as the development of Google gadgets through the JISC DOULS project.

The next steps are to build in the marketing knowledge that we have received through consultancy reports on segmentation which can help us plan out which personas we particularly want to target, and  secondly to get areas of the OU to adopt sub-set of the personas and ensure that they refresh them to keep them relevant. We already have some success with this since Student Services have adopted a persona approach to describe the “targeted services” which they want to provide through StudentHome the OU Student portal.

I can’t stress enough though how important it is to have a single coherent set of OU personas. The power comes from system developments being mapped holistically i.e. when values are shared across the organisation about meeting specific user needs and creating, buying or customising systems to meet those needs.

Agile Ballooning

Over the next three months there are projects planned which will use 140% of the available resources. In reality this means that we’ve got to hire in freelancers and contractors to cover work and ensure we’re meeting all the commitments. All of the work is strategically significant and high priority, and I’m acutely aware that the funding available to make these things happen is only available until August is unlikely to be available again in the next couple of years so we have a small window of opportunity to get things done. I thought that I’d explain what the developments are and how we’re planning to manage them.

Firstly we’ve had a number of successful small bids by academic colleagues. One of the most interesting of these, led by Doug Clow, is the development of a community based version of iSpot – this project is called iSpot local and is JISC funded for the next six months. The project is around community engagement as much as technology and there are a number of Bio Blitz’s to engage locals. Because the system is can be largely standalone we’re using a freelancer to carry out the work and using a series of hooks to the main iSpot service. The plan is that the iSpot local modules will be made available and can be set-up by anyone using a generic Drupal instance. Then you request an API key to allow your service to connect to iSpot and transfer the data between the services. Without the key you’ll still get the local community toolkit but it only becomes really useful when you can overlay all the data, the ‘spots’, that are localised to your community. There will be a map which will be set to your region or area (you’ll need to configure it initially to set it for your region) and the sightings in your area will then be displayed. The freelancer is having to work pro-actively on a steep learning curve to get the services working efficiently but he has been making good progress and we’re confident that the system will be ready for the first Bio Blitz on the 21st May.

A second project which we’re currently involved in is around the aggregation and presentation of Digital Scholarship data, this is led by Martin Weller, this is interesting because it is getting data feeds from a number of existing services and pulling data to form a view of an academic’s digital profile. As Martin says

Boyer defines scholarship as being based around four functions: discovery, integration, application, and teaching. We can think of digital scholarship then as the changes in all four of these that are brought about by the impact of digital and internet technologies. For example, if we take ‘discovery’ to be largely synonymous with ‘research’ then a digital scholarship view would be interested in the way researchers are collaborating using new technologies, sharing and visualising data, forming research communities using social media, etc

Up to now we’ve been using a contract developer (Richard Greenwood) to build the service but Richard is now required on another project around developing android apps (see below) and so we’re employing a contract developer to complete the final phase of work which is around adding further data feeds and working with the researchers to develop the visualisations.

The third project we’re undertaking is to develop mobile apps for the iSpot service. iSpot recently passed the 10,000 user mark and so it’s at a stage where we’re considering the use cases and the baseline activities that need to be developed through to ‘production’ level service. I’m excited by the opportunity to work with the OU’s Knowledge Media Institute to create mobile apps. We’re concentrating on developing an app for Android and iPhone initially but we’re creating an API which can be used across mobile platforms. Richard Greenwood is going to be working on the API and also developing the beta iPhone app. The apps will allow users to easily use their phones to capture images and upload and share them. Because of the location specific information and the visual aspects of iSpot it is a perfect service to deploy as a mobile app and it will interesting to explore how things like the image carousel and mapping information can be recreated through an app.

The fourth project we’re working, led by Mary Thorpe, is called PePLE, the concept is to create a professional working environment to support social workers, as Mary says…

PePLE is a resource for the training and continuing professional development of social workers that can be used with flexibility to fit in with the operational demands of workplaces. The resources can be used to support independent study or existing employer led provision.

The site is unique in terms of the tools it provides and we had to make the difficult decision (as we did with Cloudworks) to use a framework other that Drupal for it because there were too many constraints within Drupal to allow it to be considered. By the way there’s a good blog post about the decision to move to CodeIgniter by another of the developers, Juliette Culver, for those that want to explore the pro and cons of different PHP framework environments. The work on this project is almost complete and the site is being used as a resource to support OU Health and Social Care courses.

The fifth project we working on is the development of the OU Media Player, which I’ve already blogged about in a previous post. Work is going to plan on this project but we’ve had to be very agile and flexible in our thinking and do some reassignment of work throughout the project, to ensure things get delivered as expected since there are fours units all working together on this project and the timescales are tight. There an early demonstration of the embed service in action (note not yet accessible!).

So these projects are all significant and all have deadlines of end of July. What are the lessons we’re learning for doing all these things together and with limited resource.

1. It’s the people who make the project work. Good people who are flexible in their thinking and are into solving rather than creating problems.

2. Being pragmatic. Don’t think you can do the ‘gold star’ service within three months. In particular getting a realistic scope and keep it realistic. The developers need to be good as deciding when the requirements come in how to manage those against existing ones.

3. Use freelancers and contractors where you can on the commodity items.

4. Keep reporting and documentation to minimum. Rely on the developers to self organise, using tools such as TimePanic to keep active tracking of their time on projects.

5. Organise meetings effectively. In the team we have a system which I’m exploring where we only arrange meetings on Monday or Tuesday and the rest of the week is purely for programming. This allows the ‘flow’ that is needed to develop. So all the team only arrange any project meetings on the Monday or Tuesday in any week. The meetings are kept short and a fixed agenda. I have monthly 1-2-1’s to check progress (the team use Google Docs to record their progress for these 1-2-1’s) and rely on project blogs to keep me informed of any day-to-day changes.

6. Issues are flagged up and recorded and resolved quickly and without fuss or blame. We assign an owner to problems and fix a date to have things resolved. We use the same Drupal module that Drupal.org use for managing bugs and feature enhancements. I bring in project support staff to assist me with organising resolutions if they involve multiple parties/units.

7. Ensure the developers are not ‘managing other people’s monkeys’, I use the 1-2-1’s as an opportunity to explore what the developers spend time on to try to ensure that the majority of time is on the development of the services and not on administration, design, support.

8. Be transparent. Keep active communication channels open with the project team, stakeholders and end users. This can be done in numerous ways and will help to ensure rapid feedback and iteration.

None of these things guarantee success but they help to reduce failure rates.

Are waterfalls agile?

People talk about agile but what do they really mean by agile?

David Matthewman CIO, OU

David Matthewman CIO, OU

I read a very insightful and interesting interview with David Matthewman, the OU’s newly appointed CIO, in Computing Weekly and I’ve also had a number of discussions with him about development and programming. In the interview he says “As part of a more disciplined approach to market methodologies, Matthewman will be introducing a more prolific use of agile and scrum development, as well as service management standards such as ITIL.”

I think this is a move in the right direction for the OU and for other organisations who similarly have adopted up until recently very traditional waterfall methodologies for enterprise level system development, however agile development methodology on it’s own won’t solve the problem. I read a paper last week commissioned by Hays for example which was about agile development called “The Elephant in the Developers’ Room” – it’s kind of drawing conclusions it wants to make the case for agile, but the headline statistics alone are stunning:-

  • 60% – 80% of project failures can be attributed directly to poor requirements gathering, analysis, and management, costing US businesses $30 billion per annum
  • 50% of major projects (defined as costing >£10m) are cancelled when at least 40% of spend has been incurred
  • 40% of system problems are found by end users
  • 25% of all spending on projects is wasted as a result of re-work
  • Up to 80% of budgets are consumed fixing self-inflicted problems
  • Only 8% of large-scale applications projects (those that cost between £6 million and £10 million) succeed.
  • Just 16% of software development projects close within acceptable constraints of cost, time and quality.
  • Cost overruns of anywhere from 100% to 200% are common in software projects.
  • IT workers spend more than 34% of their time fixing software bugs

Anyway I had a twitter discussions with some developer colleagues, which is by the way the best way to solve the worlds problems, and the conclusions were as follows:-

Agile methods alone wont fix the problem of very large developments failing.

Here are some of the practical reasons why very large projects fail from experience with projects of 60m+ which I been involved in with SUN and Microsoft and others who do this stuff well on the whole:-

  1. Some issues are behavioural to do with the makeup and background of the team
  2. Some are to to do with poor management, not specific project management but people management and lack of ability to think creatively, direct appropriately and react (important as scope changes)
  3. Project scope changes, so some issues are to do with inflexibility, not reviewing scope regularly and adapting
  4. Some are to do with lack of empowerment of developers, making them both understand and grow, giving them challenge and enabling cross working
  5. Some are to do with siloing of activity, “only X knows about that” mentality
  6. lack of ownership of issues “not my problem mate” – it’s everyones problem
  7. Good (critical) people leaving during the project. Perhaps there’s a place for a ‘golden handshake’?
  8. Some are to do with complexity. i.e.not breaking the big complex system build down into smaller manageable chunks.
  9. Some are to do with people not understanding the vision. Everyone must understand it.
  10. Finally by far the best projects I’ve worked on are ones where everyone contributes to the solution, feel tied to the success of it. The reporting, logging and reviewing processes serve a purpose that those in the project understand, i.e. it’s directly relevant to assisting them and their colleagues. The organisational structure is kept light and serves to help development, rather than for MI alone.

As we move into a more commoditized, off-the-shelf, ROI, SLA, cloud-based, shared solution, outsourced, yield enhanced and recession proof world it’s important to remember that the ‘uniquness’ of an organisation is created through the innovations that come from within rather than without. Developers can still add that uniqueness to an organisation by building bespoke services very well and at scale.

We’ve just started a venture to develop the OU Media Player for example which is going to create ‘the worlds’ most accessible media player’. It’s built using existing services but we’ll add the value to make it provide captioning and accessibility services and to link to all OU media materials on a variety of platforms including the VLE. This is a very small team working over the next five months in an agile way. I’ve got 100% confidence in it’s success because it’s a great team, everyone understands how important it is to the OU and they’re being given the freedom to build it iteratively, creatively and well, i.e. serving the OU’s mission in being “open and accessible”.

Planning blight and JFDI

I was watching the gadget show the other night with Nikki and they showed a remote controlled Hummer (£37,000 car) that they were able to race around a real dirt course. The research is done by Cranfield and Nikki turned to me and said Cranfield are always doing good stuff like that why don’t you guys do good research. I of course said that we did. She said that what we should be doing for example is making a VR suit for disabled people to interact directly with their brain a nd allow them to explore VR worlds as if they had no disabilities. (read the Otherworld books by Tad Williams for the concept). I said hmm. Yes we could do that. She also said that we should rig up a system to suspend people from so that they could act out the actions conducted in a virtual world and get the total immersion experience (again Otherworld). I said Hmm. Yes we could do that.

I spoke to Patrick about this yesterday. I’m not sure about the whole VR thing but it would be good to do some work exploring hacks we could do with the Wii mote for example, by doing some ‘mashing’ of various technologies. I’ve always been a fan of this type of string and glue approach to developing stuff. He was talking about colleagues that have built the equivalent of Microsoft Surface for under £100 by using a table, projector, tracing paper and a good bit of software. I was saying how of late there is a level of project amanagement to everything we do that makes delivery take long and be less satisfying that the agile approach that we used to adopt (i.e. a clever academci and a clever devloper working together doing seat of the pants iterative prototyping). We are doing that with Social:learn so I’m doing some direct comparison and I can see how much better that approach can be with the right people and a good set of support tools. (we use Skype, Pidgin/Jabber for IM, Twitter and a Wiki to help share development stuff). Patrick said it was time we created the JDFI. “Just F*ing Do it” group. I really think we must escape to bonds of planning blight that can occur during a restructuring process and instead JFDI. That’s my main objective this year.