How Productisation Transforms the Value of Public Sector Data

August 29, 2023

Data is often only used by the team who created it. Others don’t know if it exists and, therefore, cannot use it or think about its usefulness. If we present our data sets correctly, it becomes easier for others to discover, understand and use – just like a product.

Public Sector organisations collect vast amounts of data from surveys, administrative records, transactions and monitoring. This data has significant potential to benefit the public sector, commercial organisations, researchers and even the public. However, to realise its full potential, we must go beyond simply making data available to proactively encouraging its reuse. Presenting and promoting data sources as if they were products is crucial to this transformation.

Data productisation is the process of preparing and presenting data to make it more easily discoverable and understandable by other individuals, teams, departments and organisations. It means those who collect and use data for one purpose or service invest additional effort to make the data valuable and visible to others.

Why should public sector organisations productise their data?

When we transform raw data into value-added products that are easily discovered, understood and used by other users, teams and organisations, we help to realise the full potential of data. If all kinds of users can access accurate and timely data, they can make better-informed decisions and explore the possibilities of combining it with their own data. Productised data can be made available to all types of users, stakeholders, businesses and the public, increasing transparency, trust and accountability.

Better services and innovation

More accessible and better-understood data can support innovation and growth, with collaboration between public sector organisations, private businesses, academic institutions and even the public. For example, can we leverage historical data to anticipate future needs, shape new products and services that benefit multiple, or use fast-moving data sets to power predictive insights?

Measuring progress against strategic objectives and policy intent

Bringing together data into a centralised view at a domain or industry level also means monitoring overall progress against organisational goals and more holistically understanding how effective specific initiatives are performing against KPIs is easier. This was a key consideration in our work with Ofgem, which you can read more about here.

Delivering cost savings

Promoting the reuse of existing data sets will benefit many government departments and public sector organisations. With richer and more consistent information, we can make better-informed choices more quickly, explore trends, leverage predictive analytics, explore utilisation, demand and resource allocation, drive cost savings and improve service levels.

Why isn’t this happening already?

The reality, of course, is that it is happening. However, it’s the exception rather than the default approach. It’s usually done in the context of a single service or department and by those with personal knowledge of the available data. The opportunity for data productisation is to unleash the potential of our data sets, going beyond word of mouth and accidental discovery. To realise the benefits, we must be proactive in how we present and promote the data available.

Often the quality of the data set is a blocker to it being understood and used by others. Many public sector organisations experience data quality issues, with silos, inconsistent formats, and a lack of governance. There can be a lack of the resources and investment needed to prepare data for use by others.

Work with procurement and delivery

The way digital services are procured and delivered can contribute too. Productising data so it can be reused is effectively a ‘greater good’ with all kinds of stakeholders benefitting. However, the scoping, delivery and ROI analysis of proposals typically penalises bids that recommend investing in work, technology or resources that create additional value that isn’t a direct outcome of the project under consideration. Such initiatives could deliver substantial value to the organisation - and even society as a whole - if a wider frame of reference is adopted.

Data productisation and data sharing must become a focus for public sector projects. Leaders must mandate that data be collected, stored and processed in ways that support sharing and reuse.

Find and empower data sharing

Public sector leaders must provide delivery teams and procurement managers with a mandate to select proposals that deliver additional value outside the service, department, or private sector. Providing data leadership and a vision for unlocking value from data is also crucial. There is a vital role for people to challenge the culture in organisations regarding openness and data sharing. It seems that data is often viewed as a proprietary asset rather than a public good.

Public sector organisations must implement clear and robust governance frameworks for sharing and collaborating around data. This might involve drafting template data-sharing agreements, setting standards for the format and presentation of data, and defining ownership and access policies. The emphasis needs to be on making openness the default approach embedded in the culture of public sector digital services.

Be aware of the technical challenges

There can also be technical challenges to productising data, especially when dealing with large or complex data sets. There can be legal or regulatory constraints around data collection, storage and sharing. Again, investment in the leadership, infrastructure and tools to support integration, analysis, visualisation, discovery and surfacing is critical. Very few public sector organisations have full-time, fully staffed data teams to productise and maintain their data sets.

Provide incentives and build trust

Trust is crucial for encouraging data sharing. There should be transparency about how data is collected, stored, processed and used. This ensures robust privacy and security, anonymising data where appropriate, enabling others to understand the data’s provenance and assess any biases or omissions.

To build momentum, public sector organisations should incentivise and publicise individuals, teams and departments that embrace data sharing and productisation. They also need to measure, monitor and communicate the benefits of data sharing with stakeholders, showing how it improves service delivery, informs policy decisions, and supports innovation.

Foster data partnerships

Private businesses operate public services and, therefore, often collect valuable data. Stakeholders need to focus on establishing data sharing from the start in these situations, establishing rules of engagement as well as technical arrangements to collect, store and manage data from third parties. Where practical, putting agreements to share data from the start would make it much easier to gain consistent insight from the beginning. We explored this domain in detail in our work with OZEV, which you can read more about here. 

How to productise your data sets

Set the scene

Establishing the rules of engagement and providing direction is crucial, giving individuals, teams and organisations a mandate to invest time and effort in sharing and collaboration. Public sector organisations should develop a product vision that clearly defines their mission and intended outcomes, illustrating the audiences, use cases and benefits of productising data sets. It’s also important to share a roadmap for the milestones, deliverables and timelines for creating data products, giving consideration to the resources needed to deliver a data product.

Establishing expectations for describing the features, functionality, formats, provenance and use cases for data sets is essential to encourage good practices. It’s also vital to publish guidance and requirements for product documentation, user guides, technical documentation and ‘product marketing’ literature. These elements should be considered subject to iteration and improvement, whether based on user feedback or product metrics.

Anticipate future needs

Making the best use of data for decision-making, delivery and policy also involves proactively identifying data that will be valuable. Start by identifying the data sets that are relevant and valuable to stakeholders (such as departments, public sector organisations, research and academia, and public and private sector organisations), assess future needs and policy priorities and might create a data productisation roadmap.

The next step is to assess the quality of the existing data sets and establish the extent to which they are accurate, complete, reliable and timely. Typically, this means defining data quality metrics and validating and cleaning data. This can then be monitored and evaluated on an ongoing basis, feeding into your data governance framework. Data governance covers access, storage, security, privacy and compliance, aligning with legal and regulatory requirements such as GDPR. We must also consider the vital issue of bias that may be present, a topic we feel strongly about at Valtech and have written more about here.

Create open, shareable data products

Preparing and documenting data sets beyond the needs of a digital service requires a commitment to the following practices and processes.

• Prioritise data usability, ensuring it’s open, shareable and easy to understand. Delivering this level of quality means proactively developing clear data definitions and applying metadata. Typically, that involves providing documentation and creating agreed data format standards for teams and organisations.

• It’s crucial to involve stakeholders. The reality is that they need to help change the culture and practice around data creation. This may include research and consultation to explore possibilities and gather feedback.

• Departments and organisations need to develop data-sharing policies and agreements. Beyond open data sharing and reuse, this provides individuals and teams with the framework they need to be comfortable and confident in what they do with data concerning security, compliance and governance.

• Making data easily sharable and encouraging reuse is founded on adhering to open standards, APIs and metadata standards. Additional development work to process and format data will likely be required. However, effective sharing, discovery and reuse are impossible without some work.

• Ensure that data is protected from unauthorised access and use with robust privacy measures and access control. Where appropriate, shared data should be anonymised, and there need to be regular security audits to ensure confidence and trust.

• Commit to data-driven decision-making based on the data that is being shared and analysis of service delivery and outcomes using published data sets. Document and share examples of how data is being used and the impact that it’s having.

Give the data meaning

Meaning and context are essential for promoting data reuse, especially involving business users and decision-makers. Explaining the data, its potential and how it might be used is critical to inspiring other individuals, teams and organisations to reuse it or combine it with their own data to gain insight, develop solutions or improve services to end-users.

• Ensure the data publishers clearly define the scope and purpose of their data set. This should include identifying target audiences who might benefit, detailing use cases and the intended outcomes the data set was created to deliver.

• Using standard data formats and models enables others to explore and reuse. Compiling data dictionaries in a standard, easily understood is also a meaningful way to make the data accessible to others.

• Metadata (data that describes the data set), like the source, format and structure, is crucial to helping other users, teams, and organisations understand the data and its context. Create and maintain an agreed baseline for this information.

• Bring the data to life by sharing the user stories and user testing about the needs, goals and behaviours involved in creating the data set. Similarly, document the product vision and documentation from the service or application generating each data set so that others can better understand the background and context.

• Provide examples of how the data product is being used and the insights that its creators (or others) are gaining from it. This helps to illustrate not just what the data is about but also its potential uses.

Realising the potential of public sector data

The wealth of data around public services has incredible potential for powering innovation, efficiency and improvements. Realising these benefits requires time, effort and enablement across teams, departments and organisations. Valtech consistently looks for opportunities to contribute to the greater good when we respond to public sector tenders. However, we know that collaboration and commitment to data sharing and reuse have the potential to deliver fundamental transformation.

Could we help you with your journey to leveraging the vast potential of your data? Perhaps you’re just starting your transformation, or maybe you’re part way through? We offer a complimentary discovery session to explore your next steps.

Jonathan Hale, UK Data Director, Valtech

Jonathan Hale is an accomplished data specialist with over 20 years of experience in data warehousing and migration technology and expertise in complex software and human environments. At Valtech, he leads the Data Architects and Data Engineers, providing Data Platforms, Data Engineering and Data Integration for Valtech’s EMEA customers. He is passionate about the role of data in delivering transformative change.

Data Productisation Leaders

• The National Underground Asset Register (NUAR) is envisaged to deliver at least £350 million per year of economic growth through increased efficiency, reduced asset strikes and reduced disruptions for citizens and businesses.

• Find Transport Data is a metadata catalogue that is initially focused on roads data. The aim is to continue to develop it as a resource of public and private sector transport data sets with metadata provided by the data publishers and a particular focus on geolocation information.

• The Energy Data Map has been created with the principle that “the first step to changing the energy system is to map it”. It collects data products that help others to understand energy distribution and transmission so they can play a part in redesigning the energy system to help reach net zero.

• The DfT StreetManager includes documentation of APIs and access to open data concerned with the management of roadworks. There is also a regular email newsletter with updates about what is being added to the data product and how it’s being used – a great piece of marketing to promote the data product.