How QDAP aggregates and creates new data

This use case demonstrates how the Queensland Data Analytics Platform (QDAP) can create new datasets by aggregating multiple sources of data. 

The flowchart provides an overview of the steps required for each party to fulfill their obligations throughout the process. The flowchart is limited in its focus and surrounding steps or information can be found in the workflow below. The workflow tables provide practical information which will support decision makers throughout the data sharing process. This work aims to provide data custodians, recipients and users with clarity around what they need to do when sharing and using data.

Workflow

QDAP engaged and gap identified

QDAP first identifies a gap in COVID-19 data.

ActivitiesSupporting documentsResponsibilities

QDAP identifies gap or opportunity in COVID-19 data holdings.

If data needed is currently not held by QDAP, QDAP identifies which agency holds the data.

QDAP notes any previous data sharing activities the owner/agency has undertaken.

QDAP reviews legislation and/or MOUs supporting data sharing with the agency.

QDAP identifies any restrictions to making a request to the agency.

Information sharing authorising framework (ISAF)

Privacy Impact Assessment (OIC)

Agency specific legislation

QDAP to determine if they have capacity and expertise to link, aggregate, and/or analyse the data requested.

QDAP to clearly describe the purposes for which the data from the owner/agency could be used by QDAP.

QDAP may then prepare a request for data for a specific purpose to send to the agency or owner of the data. For example, QH data on Queenslanders under quarantine due to domestic transmission, in order to aggregate the data with similar data from NSW to identify common transmission patterns.

ActivitiesSupporting documentsResponsibilities
QDAP addresses privacy and security requirements in arrangements proposed to the agency.

QDAP completes an automated request for the data with specific detais about the data requested, the purpose for which it will be used, and information about how QDAP will:
- monitor and audit the data
- control access to the data
- provide options for linkage and creating new datasets
- publish details of the dataset to their catalogue and/or a whole-of-government catalogue
- potentially publish outputs (e.g. visualisations) to the COVID-19 website
- manage recordkeeping and destruction
- potentially publish de-identified and/or aggregated datasets to Open Data  

Data catalogue of QDAP holdings including permitted usage conditions

Data catalogue of Queensland Government data holdings including custodian details

Open Data catalogue

QDAP must ensure the agency from which data is sought clearly understands what QDAP plans to do with their data.

QDAP needs to explain the kind of data required, including the necessary quality, accuracy, timliness, and format.

The Data Governance Body is the escalation point to resolve issues around permitted usage.

Supplying agency to consider request

The agency receiving the QDAP request determines whether it holds the requested data and undertakes an assessment of the data request and a preliminary assessment of the relevant data that it holds.
ActivitiesSupporting documentsResponsibilities

The Agency from which QDAP requests data undertakes a review of data holdings and determines: (1) if it holds the required data, (2) who the custodian is and, (3) who has authority to release the data.

The agency then undertakes a preliminary assessment of the data, analysing:

  • quality
  • accuracy
  • timeliness
  • format
  • effort and/or time to extract
  • security classification
  • personally, identifiable information.

Agency Information Asset Register

Qld Government Information Security Classification Framework (QGISCF)

Information quality framework guideline

The agency receiving the request from QDAP determines whether the data is available AND whether the use proposed by QDAP meets the permitted usage conditions. The agency may identify additional data sources if required.

The information asset custodian provides detailed information regarding the dataset and its condition and advice as to whether it is fit for purpose (i.e. if it can meet the requirements of the request).

Authorisation of data sharing

The authorising officer from the agency from which QDAP has requested data evaluates the request against the preliminary assessment and decides whether the data should be released.

ActivitiesSupporting documentsResponsibilities
If the request is approved, the authorisation is produced and communicated to QDAP. The authorisation includes:
- statement of authorisation and the legal instrument being relied upon (e.g. the NPPs and IPPs in the Information Privacy Act 2009 (Qld), the Public Health Act 2005 (Qld) or the Hospital and Health Boards Act 2011 (Qld))
- timeframe
- purpose of sharing
- permitted usage conditions
- access limitations
- security classification
- availability of the data and its outputs
- process for establishing new purposes
- monitoring and auditing requirements
- record keeping and destruction protocols

If the request is denied, the refusing agency must provide a detailed reason for rejecting the request.

If denied, QDAP will need to return to the identify and request stage in its search for the desired data or escalate the decision to the Data Governance Body if dissatisfied with the reasons provided.

Information Sharing Authorising Framework (ISAF)

Master sharing agreement

MOU

DG Group: if request is rejected, this group may wish to review reasons and respond to the agency’s reasons for rejection.

Agency authorising officer: ensures that due process has been followed and that the data is suitable for release. This may require additional advice for some sensitive datasets from agency legal, privacy or security officers.

Access and use of data

The supplying agency and QDAP must determine the mechanism for data to be provided to QDAP. For example, QH data custodian supplies requested data to QDAP via a secure file transfer as per the timeframe(s) in the permitted usage conditions.

Activities Supporting documentsResponsibilities
Agency supplying the data to QDAP determines the mechanism for the data to be provided to QDAP based on:
- its security classification
- the sensitivity of the data
- its format

Data is then shared with QDAP at the time and in the format agreed upon.

Authorisation - permitted usage conditions *

Qld Government Information Security Classification Framework (QGISCF)

Information exchange schedule

Information asset custodian arranges transfer of data via email, a web form, an MSA or an MOU, and ensures that timeframe conditions are met.

QDAP receives data and stores / restricts access in accordance with permitted usage conditions. QDAP stores authorisation along with the data.

Data Governance Body periodically audits to ensure authorisations and permitted usage conditions are being met.

Once QDAP has received the data, it reviews it and ensures it is fit for purpose.

Activities Supporting documentsResponsibilities
QDAP is to review the data and ensure it is fit for purpose. Depending on the state of the data supplied and the requirements of the request, this may involve activities such as quality assurance, data cleansing, and applying unique identifiers for data matching.

Original data request

Permitted usage conditions in the Authorisation *

Any additional conditions of use set by the supplying agency

QDAP to review and prepare data for aggregation/analysis. This may require QDAP to liaise with the supplying agency to resolve issues.

Data Governance Body functions as an escalation point to resolve any issues with the supplying agency if required.

QDAP can then use the data for its intended purpose. For example, data is used to map similarities or differences in patterns of community transmission for COVID-19 communications in NSW and Qld.

Activities Supporting documentsResponsibilities
This step may involve data being used and accessed within the confines of QDAP as well as QDAP processing data and providing it to other Qld Gov agencies or interstate agencies to use or consume. Original data request

Permitted usage conditions in the Authorisation *

Any additional conditions of use set by the supplying agency

Information access and use policy
QDAP aggregates and/or analyses agency and other data as per the original request.

Data users are made aware of the permitted usage conditions and use data only for its intended purpose(s).

The Data Governance Body has oversight of the process, reviews outcomes, makes suggestions for improvements and deals with any issues escalated to it.

* To assist with the protection of data and information shared to support COVID-19 related work, a Permitted usage authority register has been established. Queensland Government users can search this register to find out the permitted usage conditions of data and information shared through the Data Catalogue.  It can be found at the Queensland Government Data, Information Sharing and Analytics collaboration site. For access, please contact transformation@qgcio.qld.gov.au.

Data management and governance

Appropriate record keeping and publishing practices must be followed.

ActivitiesSupporting documentsResponsibilities

QDAP ensures that authorisations are kept for recordkeeping purposes and available for auditing.

QDAP publishes to appropriate and agreed repositories in accordance with the conditions specified in the original request and the permitted usage conditions agreed with the agency.

Public Records Act

Records Governance Principles

Open data policy statement

Information access and use policy

The Data Governance Body provides oversight and review of the QDAP process. It deals with escalated issues, such as use of the data for purposes outside the original agreed purpose.

Access to data must be logged and monitored.

ActivitiesSupporting documentsResponsibilities
QDAP has documented/automated processes in place to ensure that only those permitted to access and use the data have access to it and adhere to permitted usage conditions.

Information access and use policy

Audit schedule

There should be access notifications at log in to ensure users are aware of inappropriate access and use penalties (as per CCC Operation Impala Recommendations)

Requesting agency provides permission for their staff to access data prior to requesting access to QDAP.

QDAP ensures appropriate user access permissions are provided and kept up to date. It also ensures users are aware of their responsibilities.

Data Governance Body provides oversight and review and deals with escalation of issues.

There is a body in place to deal with escalated issues and governance.

ActivitiesSupporting documentsResponsibilities
There is a Data Governance Body or Group in place to oversee the process, deal with escalated issues, resolve problems and make determinations. Ideally this body includes representatives from key data supplying and requesting agencies with the seniority and/or authority to act. Data governance charterThe Data Governance Body's responsibilities would include, but are not limited to:
- resolving unforeseen issues
- deciding on access requests
- responding to rejected requests for data 
- following up requests for data not met as requested (e.g. quality, timeliness, etc.)
- negotiating and amending conditions of use
- agreeing to or rejecting additional permitted usage conditions
- authorising the use of additional data sources
- dealing with adverse audit results (e.g. unauthorised access, use or release)

At the end of its life-cycle, the data is to be retired or destroyed appropriately.

ActivitiesSupporting documentsResponsibilities
Once the permitted usage timeframe has ended, the data should be destroyed unless other arrangements have been made.

Permitted usage conditions from the Authorisation

Public Records Act

Records Governance Principles

Retention and disposal schedule

QDAP to regularly review data holdings and determine if data has reached the end of its retention period.

Supplying agency to request or approve destruction or provide reasons for retention.


Last Reviewed: 09 June 2020