Tag Archives: open data

Making use of Open Data

Darllenwch y flogbost yn Gymraeg

The Wales Audit Office recently released our first Open Dataset. What happened next? Ben Proctor of Open Data Institute Cardiff talks us through how he made use of the data.

A screenshot of a dynamic map created by Ben Proctor to show levels of Council Tax per head of population in Wales

A screenshot of a dynamic map created by Ben Proctor to show levels of Council Tax per head of population in Wales

Oooo! new data

I was excited to see that the Wales Audit Office had released a set of data as open data. Open data is data that anyone can find access and use and it is the most useful sort of data.

Dyfrig Williams wrote about the process they’d gone through to release this data set (a summary of the audit data from each local authority in Wales for each year). The data is a simple table and you can download it as a CSV file (essentially a file that will work in any spreadsheet programme) here.

But there are problems

I downloaded the file and quickly spotted some problems. These are not errors exactly but just things that are missing or inconsistent and will make some uses of the data a bit harder. But this is not a complaint, because one of the attractive features of open data is that I could resolve these problems. I can do this because the Wales Audit Office have released the data under the Open Government Licence. This tells me I don’t need their permission to do anything with the data and there are no limits to what I can do with it (apart from I have to make it clear where it came from).

I can fix the problems

These are the things I did to my copy of the data.

I changed the format of the “financial year column” because in the Wales Audit Office file some of these are numbers and some are text.

I added a column of GSS codes. GSS codes are codes that are used to identify local authorities (and other boundaries). Having the GSS code means you don’t have to worry about whether the data says Anglesey Council, or Isle of Anglesey Council or Ynys Môn. And with the GSS code I could add “polygons” for each council. Polygons are basically instructions on how to draw the outline of each council and information about where to put the drawing on a map.

With these changes I was able to draw a series of maps showing the level of council tax per head in each local authority and how this has changed over time.

And given the Wales Audit Office an improved file

And I’ve been able to hand back to the Wales Audit Office a KML file. This is a file suitable for use in mapping software. Anyone who wants to visualise the Wales Audit Office data on a map can just open the KML file and get going.

You can download this mapping file yourself.

Why did I do this?

I’m part of the core team at ODI-Cardiff so I get excited about open data.
It took me a very few minutes.
I’m trying to get better at using a Google service called Fusion Tables and this is a good opportunity to experiment.
I’m actually quite interested in what this data might tell us.

Releasing Open Data

Darllenwch y flogbost yn Gymraeg

How can organisations release Open Data? Dyfrig Williams looks at the process of making the data behind the Wales Audit Office’s Local Government Financial Statements report freely available, and shares the dataset at the bottom of this post.

I’ve previously blogged on how the Wales Audit Office is looking to challenge our existing use of data and technology as part of the Cutting Edge Audit project. My role on the project has been to look at how we acquire data.

How does Open Data fit into acquiring data?

The below diagram shows the rationale behind my work. For me, we need to share the data that we have in order to develop our relationships with our client bodies so that we can gather data effectively. Part of this is about “being the change that you want to see.” Auditors are sometimes seen as being risk averse, but in the Good Practice Exchange we’ve seen that when we work differently, we enable others to do the same. A number of local authorities have reported back to us about how they’ve been able to challenge the limitations of the websites that they’re able to visit and the social media that they can access because of how we share knowledge. By making data openly available, we can demonstrate that there is little risk, as long as the process is well managed.

A cycle illustrating that by open data results from how we share data and how it feeds into acquiring data

As I mentioned in my original post, finding an appropriate dataset was more challenging than I thought that it might be. As we often don’t have the right to share the data that we collect from clients during our audits. However after a bit of research, we found the data behind the Local Government Financial Statements report, which is a report on local government bodies’ accounts. This was safe data to release because it’s already available on each council’s website as part of their accounts, but we are the only organisation that collates this data. The data within the report is analysed on a national basis, but by releasing the dataset we can enable councils and other interested stakeholders to look at the data on a county by county basis and to compare and contrast their accounts against others. The data is used by the Wales Audit Office to support local audit work and for general benchmarking. The report itself looks at the quality of accounts, and is based on the data that’s released before amendments – we don’t keep track of prior-period adjustments.

How did we go about making this data open?

Our starting point was a spreadsheet that we use internally that contains the datasets dating back to 2008-09. We were a bit disappointed to learn that the requirements for local authorities to provide this data in this structure has now changed, so there won’t be comparable data available next year. However, this dataset served as a good test for a future approach. In the longer term it would be worth us looking at how we could make continuous data available in order to reduce the burden of reporting requirements. Lucy Knight from Devon County Council has a really useful example that we can draw on in her lunchtime lecture for the Open Data Institute on Making open data happen in local government

We used Hendrik Grothuis’s post on making data open and the Open Data Institute’s Consumers Checklist as rough guides for the process. Our first step in cleaning up the data was to look at which data was ours to share, and which data was already available from other sources. We decided to remove the data that was already made available through StatsWales to avoid duplication, but should you want to think about using this dataset with some of the ones that we used internally, these may provide a good starting point:

We then used CSV Lint to check whether the file was readable. We were pleased to discover that we had a valid file, but we also found ways that we could improve it. We turned the dataset around so that the data items go horizontally and the years go vertically. We also created a null value to indicate where the data was unavailable. A quick Google search was enough for us to discover how to note empty cells.

As a Welsh public sector organisation, we are required to make the data available bilingually, so we sent it to the translators to make sure that we got each technical term exactly right.

Publishing the data

When it came to publishing the data, we decided to publish it as part of this post on the Good Practice Exchange blog. It would be a very lonely looking dataset on an Open Data platform at the moment, but the hope is that we can identify other datasets that we can release going forward. We looked at potential platforms that could be used, including open source options like CKAN and DKAN (both of which would integrate with our Drupal Content Management System), as well as cloud based platforms like Socrata. As an organisation we’re moving to the cloud when it makes sense, but there may be things that we could learn from Audit Scotland’s Innovation Zone, which has been set up to allow their staff to test new software and platforms in a lightly regulated space. This gives staff the opportunity to test new ways of working.

As per our recent webinar on Open Standards, we’ve chosen to publish the data in CSV instead of a proprietary format like Excel. This means that it can be used by a wide variety of software, and hopefully as wide a variety of people as possible.

It’s now up to us to ensure that this data is discoverable by tagging it effectively, and we will also publicise the dataset through the networks that we’ve built through our prior work on Open Data. Our next challenge is to track how the data is used, so if you do use the dataset, we’d love to have your feedback about the format and what you used it for.

Learning from the Welsh Government

The Welsh Government were a great help throughout my work on the Cutting Edge Audit project. They shared learning from their approaches, and we also attended meetings together to learn more about Cardiff and Monmouthshire Councils’ approaches. It was fascinating to hear that the Welsh Government’s own staff use StatsWales to share and gather data as it’s open and transparent. This is something for us to think about in our own journey forward – how we can make data more accessible for both internal and external stakeholders.

We ended up using the Welsh Government’s approach to Metadata as a template for our own work. Metadata is a set of data that describes and gives information about other data, and it’s really important because it gives context around the data that is being shared. You can find the metadata at the bottom of this post alongside a link to the data itself.

Feedback

Your feedback on our approach here is really important. As this is an initial test of how we might make data open and shareable, your feedback will be used to shape how this might progress. As an organisation, we’re very keen to look at how we can make better use of data to help public services improve, and also to walk the talk in terms of our own digital practice. The Auditor General for Wales talks about enabling innovation through well managed risks before every one of our shared learning events. We’re looking to share our own learning so that people can learn from our experiences, be they good or bad. We always say that there’s no point reinventing the wheel. By working openly and transparently, we hope that organisations can build on what we’re doing so that they can share data as effectively as possible in order to improve the services that they provide.

Dataset: Local Government Financial Statements

Metadata

Title Data from Local Government Financial Statements 2008-09 to 2015-16
Last update 01/04/2017
Next update No longer updated.
Publishing organisation Wales Audit Office
Source Local Government Financial Statements 2008-09 to 2015-16
Contact email good.practice@audit.wales
Lowest level of geographical disaggregation Local authorities
Geographical coverage Local authorities
Languages covered English and Welsh
Data licensing You may use and re-use this data free of charge in any format or medium, under the terms of the Open Government License – see http://www.nationalarchives.gov.uk/doc/open-government-licence

Glossary

Acquiring data for a cutting edge audit office

How is the Wales Audit Office working to ensure that it provides audit that’s fit for the future? Dyfrig Williams blogs below on his work with the Cutting Edge Audit project.

Over the past few months, I’ve been working on the Cutting Edge Audit project, which looks at how the Wales Audit Office can challenge our existing use of data and technology and assumptions that we normally take for granted. We’re thinking radically about how we might use new technology to transform the way that we work.

It’s been a fantastic piece of work to undertake, which has really put that radical thinking into practice. The project’s being led by my colleague Steve Lisle, who is reporting directly to the Auditor General for Wales. This has meant that we’ve moved away from hierarchy into a much flatter structure. We’ve also been outcome focussed – we’ve been testing and prototyping as we go so that our risks are well managed and that we learn from failure.

I’ve been working on how the Wales Audit Office acquires data to give us deeper knowledge and fresh insight.

Data Maturity

data_maturityIt was helpful to think about Data Maturity when we were doing this work. Data Maturity is the journey towards improvement and increased capability in using data. Data Orchard have used this model (which I’ve nicked from a great post by Ben Proctor) of stages in an organisation’s development:

  1. Ad-hoc gathering of data in some areas
  2. Pulling data together centrally
  3. Starting to use data looking backwards
  4. Using data in real time to manage the organisation and move resources rapidly
  5. Modelling the future before making decisions to enable better decisions to be taken
  6. Modelling the future the organisation wants and working backwards to understand what needs to happen now to deliver that future

This is very much a journey for us as an organisation, but it has helped to inform my thinking. It’s helped me think about how we get to point 6, where we’re modelling the future that the organisation is working towards, and ensure that the things that I’m working on set us out on the right path beyond the lifespan of the Cutting Edge project.

My prototypes

I’ve been working on two different tests within this field. The first is an Open Data prototype, which has been more challenging than I expected because the Wales Audit Office is a secondary user of data. This means that we use data that is gathered by others, so we don’t always have the right to share it. I have managed to find a useful dataset though, so my next step is to set it free into the world and look at the challenges around how we can make it as useful as possible.

I’ve been putting the Good Practice Exchange’s principles into practice in this work by visiting other organisations to learn about the work they’re already doing because there’s no point reinventing the wheel. I’ve also been thinking about how we adapt rather than adopt their work to suit our organisational needs, because after all, a one-size-fits-all approach never works.

I’ve blogged before about why the public sector needs to start thinking about its approach to Open Data, and we subsequently ran a Google Hangout to look at why it’s an important topic. Hendrik Grothius has written an excellent blogpost on how organisations can start to publish Open Data, and it will be a brilliant starting point for me as I get to grips with this.

My second piece of work has been looking at how we enable our staff to make better use of data, thereby minimising the audit burden. I’m looking at how we can bring together data from public bodies in a way that makes it easy to access, open to everyone, and give us an improved insight into the performance of the Welsh public sector, and international comparators. I’ve been talking to our staff so that I can better understand what type of approach would be useful to them. I’ve developed personas to help guide our work in this area, which will shape the next phase of this work and ensure that my part of the final report is focused on user need.

Iteration

I’ll be writing future posts to share my approaches, what I’ve learnt and what I would do differently next time. We are working iteratively so that we learn from each development and how we can build on that learning going forwards. If those prototypes don’t work, we’ll be looking to learn from failure and see what the organisation can do differently in the future.

At all our Good Practice Exchange seminars we hear that public services can’t continue to work in the same way in these austere times. It’s been great working on a practical project at the Wales Audit Office, as we’re getting to grips with those same challenges and applying new thinking to our work.

What can Open Data do for public services?

The Wales Audit Office is holding a Google Hangout on Open Data. It will look at how Open Data can help public services to deliver joined up, transparent and effective public services. In this blogpost, Dyfrig Williams looks at why the Good Practice Exchange is interested in the topic.

Last year the Effective Services for Vulnerable Groups team at the Welsh Government approached us about the possibility of doing some work around highlighting good practice around effective data sharing. When we held a scoping meeting, we found ourselves being drawn into two slightly different discussions – one on sharing personal data, and another on the merits of Open Data. To do justice to both, we decided to hold two separate events. The event on personal data really helped bust some myths around data protection, especially where the Assistant Information Commissioner for Wales got to grips with the issues that public services face.

Open Data was subsequently added to our list of events for this year. At the same time, we found out that the Office for National Statistics was interested in running a webinar the topic, so in the spirit of collaboration, we’ll be running this Hangout jointly with them.

So what is Open Data?

Open data is data that anyone can access, use or share. When big companies or governments release non-personal data, it enables businesses and citizens to make improvements to their communities. The Open Data Institute has produced an introductory video to explain some of the benefits of Open Data.

Open Data Impact have identified four ways in which Open Data is changing the world:

  • Improving government (by tackling corruption, increasing transparency and improving public services)
  • Empowering citizens (to take control of their lives through informed decision making)
  • Creating opportunity (by fostering innovation, promoting economic growth and creating jobs)
  • Solving Public Problems (by giving access to new forms of data-driven assessment of the problems at hand. It also enables targeted interventions and enhanced collaboration)

What does this mean in reality?

There are lots of case studies of where public services have released Open Data to benefit wider society. Transport for London released data that has seen a 58:1 return on investment, which has led to the development of companies like Citymapper.

In Canada, open data exposed one of the biggest tax frauds in the country’s history through an examination of the publicly available Annual Information Returns.

There are also some great examples in local government. The Leeds Data Mill have a fantastic City Dashboard, which shares open data feeds through simple graphics to give a snapshot of the city at any given moment. The Hampshire Hub has undertaken lots of work, including how public services respond to a pending weather event. It also identified GP surgeries who’ll be most under pressure due to increases in demand.

So why isn’t everybody jumping on the bandwagon?

One of the issues is around understanding. Dan Slee has written a great post on why jargon and a lack of understanding is a big issue when it comes to sharing good practice around Open Data. And if like me you’re not a particularly techy individual, it makes it difficult to apply the learning to real world public service delivery.

Organisations also need to be clear about why they’re releasing data and the outcomes that they want. What does success look like? How will public services evidence the difference it’s made to their communities? The National Audit Office produced a fascinating report on Implementing transparency in 2012 that found that “government needs a better understanding of costs, benefits and use to assess whether transparency is meeting its objectives of increasing accountability, supporting service improvement and stimulating economic growth.” I suspect that the UK Government is not alone in that.

How can a webinar help?

This webinar with the Office for National Statistics will help us to examine how Open Data can help public services deliver joined up, transparent and effective public services. We’ll be looking at:

  • What Open Data is
  • How we can make data open
  • The tangible benefits
  • The next steps

The panel will include Rob Davidson from the Office for National Statistics, Esko Reinikainen from the Open Data Institute Node in Cardiff, and Helen Wilkinson from Natural Resources Wales, who will share practical learning from the Lle Portal.

The Hangout will be very participative, so we’d love to hear from you and to receive your questions before and during the discussion. And if you’re already making the most of Open Data, we’d love to find out whether it’s helping you to empower people, improve services and make the communities that you serve better places in which to live.