How should the completeness and quality of curated nanomaterial data be evaluated?

,

The design of novel nanomaterials with desirable properties and acceptable safety profiles, as well as the appropriate regulation of both new and existing nanomaterials, relies upon nanoscience researchers (both experimentalists and computational modellers), risk assessors, regulators and other relevant stakeholders having access to the necessary data and metadata.
These data should be sufficiently complete, including their associated metadata, and of acceptable quality to render them fit for their intended purpose e.g. risk assessment. However, defining what one means by data which are "sufficiently complete" and of "acceptable quality" is non-trivial in general and is arguably especially challenging for the nanoscience area.
The current paper is part of a series of articles 9,17 that address various aspects of nanomaterial data curation, arising from the Nanomaterial Data Curation Initiative (NDCI), where curation is defined as a "broad term encompassing all aspects involved with assimilating data into centralized repositories or sharable formats". 9 A variety of nanomaterial data resources, holding different kinds of data related to nanomaterials in a variety of formats, currently exist. Many of these were recently reviewed. 9,18,19 The number of nanomaterial data resources is expected to increase as a result of ongoing research projects. 4,19 An overview of the articles planned for the NDCI series was presented in Hendren et al. 9 At the time of writing, an article on curation workflows 17 was published and articles dedicated to curator responsibilities, data integration and metadata were at various stages of development. The current paper addresses the question of how to evaluate the degree to which curated nanomaterial data are "sufficiently complete" and of "acceptable quality". In order to address this central question, the current paper addresses a number of key issues: (1) what the terms data completeness and quality mean; (2) why these issues are important; (3) the specific requirements for nanomaterial data and metadata intended to support the needs of specific stakeholders; (4) how to most appropriately score the degree of completeness and quality for a given nanomaterial data collection. The abstract meaning of data completeness and quality in a range of relevant disciplines is reviewed and the importance of these concepts to the area of nanomaterial data curation is explained. An overview of existing approaches for characterising the degree of completeness and quality of (curated) nanomaterial data is presented, with a focus on those currently employed by curated nanomaterial data resources. Approaches to evaluating data completeness and quality in mature disciplines are also reviewed, with a view to considering how the relatively young discipline of nanoscience could learn from these disciplines. However, as is also discussed, there are specific challenges associated with nanomaterial data which affect assessment of their completeness and quality. Drawing upon the discussion of these issues, the current paper concludes with a set of recommendations aimed at promoting and, in some cases, establishing best practice regarding the manner in which the completeness and quality of curated nanomaterial data should be evaluated.
The snapshot of current practice, discussion of key challenges and recommendations were informed via a review of the published literature as well as responses to a survey distributed amongst a variety of stakeholders associated with a range of nanomaterial data resources. The survey and responses can be found in the ESI, † along with an overview of the nanomaterial data resources managed by these stakeholderswith a focus on how they address the issues related to data completeness and quality. The perspectives of individuals involved in a variety of nanomaterial data resources were captured via this survey. However, the resources for which respondents agreed to participate in this survey should not be seen as comprehensive. 9,18,19 For the purposes of the survey, the Nanomaterial Data Curation Initiative (NDCI) identified 24 data resources that addressed various nanomaterial data types: from cytotoxicity test results to consumer product information. Some of the identified resources were exclusively focussed on nanomaterial data, whereas others were broader databases holding some data for nanomaterials. Representatives of the 24 data resources were contacted by the NDCI and, in total, 12 liaisons, corresponding to nine (38%) of the 24 nanomaterial data resources, responded to the NDCI data completeness and quality survey. Some of the nine resources incorporated primary experimental data, whilst others were exclusively populated via literature curation. Some of these were in-house resources, whilst others were publicly available via the internet. The median experience of the survey respondents was 5 years in the nanomaterial data curation field, 10.5 years in the wider nanoscience field, and 5.5 years in the broader data curation field.
The rest of this paper is organised as follows. Section 2 reviews the meaning of data completeness and quality, in abstract terms, and then explains the importance of these issues in the context of nanomaterial data curation. Section 3 reviews existing proposals for characterising the completeness and quality of (curated) nanomaterial data. Section 4 reviews approaches for evaluating (curated) data completeness and quality which are employed in mature fields. Section 5 then discusses the key challenges associated with nanomaterial data which need to be taken into account when evaluating their completeness and quality. Section 6 presents the recommendations for evaluating curated nanomaterial data completeness and quality.

The meaning and importance of data completeness and quality
The importance of data completeness and quality is made clear by explaining what these concepts mean and their implications for a range of important issues. (Data completeness and quality are hereafter referred to as Key concept 1 and Key concept 3, with full descriptions presented in Tables 1 and 3, respectively.) The precise meanings of these concepts and the issues with which they are related are defined somewhat differently in the varied fields which are relevant to nanomaterial data curation e.g. informatics, toxicology and risk assessment. Nonetheless, it is possible to provide broad and flexible definitions which encompass a variety of perspectives.
Broad and flexible definitions of data completeness and quality are presented in Tables 1 and 3 respectively. These reflect the different and sometimes inconsistent definitions presented, either implicitly or explicitly, in the literature, during discussions amongst the co-authors and by respondents to the NDCI data completeness and quality survey. (The perspectives of the survey respondents are presented in the ESI. † Literature definitions of data completeness 9,20-24 and quality 9,[20][21][22][23]25,26 are provided in ESI Tables S3 and S5 † respectively.) Section 6.1.1 proposes that more precise definitions be adopted by the nanoscience community. These more precise definitions are generally consistent with the definitions presented in Tables 1 and 3, but some issues incorporated into  those broad and flexible definitions are deemed out of scope.  However, the definitions provided in Tables 1 and 3 encompass the range of different perspectives encountered when preparing this paper. Hence, these definitions serve as a reference point for the purpose of reviewing existing approaches to evaluating data completeness and quality in sections 3, 4 and ESI S2. † The following discussion expands upon the broad and flexible definitions presented in Tables 1 and 3. The importance of these concepts for nanomaterial data curation, and the issues with which they are commonly associated, is explained with reference to the nanoscience literature.
Data completeness may be considered a measure of the availability of the necessary, non-redundant data and associated metadata for a given entity (e.g. a nanomaterial). (Some scientists consider the availability of "metadata" to be a separate issue to data completeness.) 20,21 The term "metadata" is broadly defined as "data which describes data" 27 or "data about the data". 28 Defining exactly what is meant by "data" as opposed to "metadata" is challenging. For example, physicochemical characterisation data may be considered metadata associated with a biological datum obtained from testing a given nanomaterial in some assay. 3 However, precisely delineating "data" and "metadata" lies beyond the scope of the current article. In this article, data and metadata are collectively referred to as "(meta)data".
Generally, data completeness assesses the extent to which experimental details are described and associated experimental results are reported. One means of assessing the degree of completeness compliance is to employ a minimum information checklist. (This concept is referred to hereafter as Key Concept 2 and a broad and flexible definition is presented in Table 2. Literature definitions 28,29 are presented in ESI Table S4. †) However, one may also draw a distinction between data which are truly complete and data which are compliant with a minimum information checklist. The checklist may simply specify the most important, but not the only important, (meta)data. For example, in the case of nanomaterial physicochemical characterisation, measurement of a large number of properties might be considered necessary for complete characterisation but not truly essential to achieve all study goals. These properties might be distinguished from "priority" or "minimum" properties which are "essential" to determine. 3 The degree of data completeness, insofar as this refers to description of the necessary experimental details and availability of (raw) data, needs to be evaluated in a range of different nanoscience contexts. Firstly, it impacts the extent to which data areand can be verified to bereproducible. [30][31][32][33] Reproducibility 32-34 is contingent upon the degree to which the tested nanomaterial is identified and the experimental protocols, including the precise experimental conditions, are described. 35 Given the context dependence of many properties The completeness of data and associated metadata may be considered a measure of the availability of the necessary, non-redundant (meta) data for a given entity e.g. a nanomaterial or a set of nanomaterials in the context of nanoscience. However, there is no definitive consensus regarding exactly how data completeness should be defined in the nanoscience, or wider scientific, community. 9,20-24 Indeed, metadata availability may be considered an issue distinct from data completeness. 20,21 Data completeness may be considered to include, amongst other kinds of data and metadata, the extent of nanomaterial characterisation, both physicochemical and biological, under a specified set of experimental conditions and time points. It may also encompass the degree to which experimental details are described, as well as the availability of raw data, processed data, or derived data from the assays used for nanomaterial characterisation. Data completeness may be considered to be highly dependent upon both the questions posed of the data and the kinds of data, nanomaterials and applications being considered. Data completeness may be defined in terms of the degree of compliance with a minimum information checklist (Table 2). However, when estimating the degree of data completeness, it should be recognised that this will not necessarily be based upon consideration of all independent variables which determine, say, a given result obtained from a particular biological assay. This is especially the case when data completeness is assessed with respect to a predefined minimum information checklist (Table 2). Precise definitions of completeness may evolve in tandem with scientific understanding. which may identify nanomaterials, these two issues are interrelated. This is because nanomaterial identification, if based on physicochemical measurements, is not meaningful unless the corresponding experimental protocols are adequately described. 3,[36][37][38][39][40] Providing sufficient (meta)data to ensure the nanomaterial being considered is identified, to the degree required, is also inherently important to achieve the goals of "uniqueness" and "equivalency". 41 Establishing "uniqueness" means determining that nanomaterial A is different from B. 41 Establishing "equivalency" means determining that nanomaterial A isessentiallythe same as B. 41 Achieving "uniqueness" allows so-called "conflicting" results to be resolved. 3 Achieving "equivalency" allows for data integration (e.g. to interrogate relationships between different kinds of data) using data reported for the same, or functionally equivalent, nanomaterial in different studies.
Physicochemical characterisation also assists with explaining observed differences in (biological) effects. 3 Indeed, it facilitates the development of computational models for (biological) activity, based on the physicochemical properties as explanatory variables. Modelling of nanomaterial effects may entail the development of nanomaterial quantitative structure activity relationships (QSARs)termed "nano-QSARs", 42 nanoscale structure-activity relationships ("nanoSARs") 43 and quantitative nanostructure-activity relationships ("QNARs") 44 or "grouping" and "read-across" predictions for nanomaterial biological activity. 44,45 Reporting of the experimental details associated with the generation of a given biological or physicochemical measurement facilitates assessment of whether data from different sources might be combined for modelling, given the potential trade-off between dataset size and heterogeneity. 46,47 Data quality may be considered a measure of the potential usefulness, clarity, correctness and trustworthiness of data. Some data quality assessment proposals 23,35,48 may talk interchangeably about the quality of data, datasets (or "data sets"), studies and publications. However, subsets of data from a given source (e.g. a dataset, study report or journal article) may be considered to be of different quality, depending upon exactly how data quality is defined and assessed. 49 For example, the cytotoxicity data reported in a publication might be considered of different quality compared to the genotoxicity data. As another example, the data obtained for a single nanomaterial using a single assay might be considered of higher quality than the data obtained for a different nanomaterial and/or assay.
Whilst the quality of individual data points is an important issue, data points whichviewed in isolationmay be considered of insufficient quality to be useful may possibly be useful when used in combination with other data. For example, toxicity data which are evaluated as less reliable might be combined via a "weight-of-evidence" approach. 35 As another example, in the context of statistical analysis, large sample sizes may partially offset random measurement errors. 50 However, the importance of the reliability of the original data which are to be combined cannot be overlooked in either context. 23,50 According to some definitions, data quality may be partly assessed based upon the relevance of the data for answering a specific question. 27,48 Similarly, data completeness may also be considered highly context dependent. Here, the specific context refers to the kinds of data, the kinds of nanomaterials, the kinds of applications and the kinds of questions that need to be answered by a particular end user of the data. In other words, the degree to which the data are complete may be contingent upon "the defined [business] information demand". 27 Table 2 Key concept 2: minimum information checklist. Broad and flexible definition employed for reviewing prior work Minimum information checklists might otherwise be referred to as minimum information standards, minimum information criteria, minimum information guidelines or data reporting guidelines etc. 28,29 These checklists define a set of data and metadata which "should" be reportedif availableby experimentalists and/or captured during data curation. Again, the precise set of data and metadata which "should" be reported may be considered to be highly dependent upon both the questions posed of the data and the kinds of data, nanomaterials and applications being considered. There are two possible interpretations of the purpose of these checklists: (1) they should be used to support assessment of data completeness (Table 1); (2) data should be considered unacceptable if they are not 100% compliant with the checklist. Data quality may be considered a measure of the potential usefulness, clarity, correctness and trustworthiness of data and datasets. However, there is no definitive consensus regarding exactly how data quality should be defined in the nanoscience, or wider scientific, community. 9,[20][21][22][23]25,26 Data quality may be considered dependent upon the degree to which the meaning of the data is "clear" and the extent to which the data are "plausible". 48 In turn, this may be considered to incorporate (aspects of) data completeness (Table 1). For example, data quality may be considered 23 to be (partly) dependent upon the "reproducibility" of data [31][32][33][34] and the extent to which data are reproducible and their reproducibility can be assessed will partly depend upon the degree of data completeness in terms of the, readily accessible, available metadata and raw data. 30,35 As well as "reproducibility", data quality may be considered to incorporate a variety of related issues. These issues include systematic and random "errors" in the data, 32,33 data "precision" (which may be considered 33 related to notions such as "repeatability" [32][33][34][35] or "within-laboratory reproducibility"), 33 "accuracy" and "uncertainty". 20,23,25,27,32,33,35,[51][52][53][54][55] (As indicated by the cited references, different scientists may provide somewhat different definitions for these concepts. These concepts may be considered in a qualitative or quantitative sense.) Data quality may also be considered to be dependent upon the "relevance" of the data for answering a specific question, although data "relevance" might be considered an entirely distinct issue from data quality. 23,48 In the context of data curation, not only the quality of the original experimental data needs to be considered but also quality considerations associated with curated data. Quality considerations associated with curation include the probability of transcription errors 56 and possibly 57 whether a given dataset, structured according to some standardised format (e.g. XML based), 58 was compliant with the rules of the applicable standardised format (e.g. as documented via an XML schema). 59 Such compliance, amongst other possible aspects of data quality, could be determined using validation software.
None of the preceding discussion addresses the key question of how exactly to evaluate data completeness or quality for (curated) nanomaterial data. This question will be addressed in subsequent sections of the current paper.

Existing proposals for evaluating nanomaterial data completeness and quality
A plethora of proposals has been presented for assessing data completeness and quality in the nanoscience area. Because it would not be practical to comprehensively list and discuss all existing proposals in the current work, the following discussion (sections 3.1 and 3.2) aims to be illustrative of the different proposals which have been developedwith an emphasis on the most recent and those which are employed by the maintainers of specific curated nanomaterial data resources. Examples are taken from the published literature as well as the responses to the survey which informed the current article. A summary of the evaluation schemes, if any, employed by each of the data resources represented by the respondents to the survey is provided in the ESI. †

An overview of nanomaterial data completeness proposals
Considerable attention has been paid to identifying the minimum set of physicochemical parameters for which it is anticipated that nanomaterials with similar values for these parameters would exhibit similar effects in biological (e.g. toxicological) tests or clinical studies. 3 Here, "physicochemical parameters" refers to the characteristics/properties relevant for the description of a nanomaterial such as chemical composition, shape, size and size distribution statistics. A number of lists exist, including the well-known MINChar Initiative Parameters List, proposed in 2008. 60 Earlier efforts to provide minimum characterisation criteria for nanomaterials included the work carried out by the prototype Nanoparticle Information Library (NIL). [61][62][63] The prototype NIL was developed in 2004 to illustrate how nanomaterial data could be organised and gave examples of what physicochemical parameters, along with corresponding information regarding synthesis and characterisation methodology, might be included for nanomaterial characterisation (see the ESI † for further details). In 2012, Stefaniak et al. identified and carefully analysed 28 lists ( published between 2004 and 2011) which proposed "properties of interest" (for risk assessment), from which 18 lists of "minimum"or, in their terms, "priority"properties were discerned. 3 These authors summarised the properties found on these lists and the corresponding frequency of occurrence across all lists. Other lists 39,64-69 of important physicochemical parameters have been published subsequent to the analysis of Stefaniak et al. 3 Arguably, within nanoscience, less attention 70 has been paid to the question of which additional experimental details (e.g. the cell density, 71 number of particles per cell, 72 cell line used, passage number used or exposure medium constituents 73,74 in cell-based in vitro assays) need to be recorded. It is important to note that many of the physicochemical characteristics which define the identity of a nanomaterial are highly dependent upon experimental conditions such as the pH and biological macromolecules found in the suspension medium. 36,39,40 Nonetheless, some lists which specify key experimental details that should be reported (in addition to key physicochemical parameters) do exist. 3,60,64,66,75,76 Indeed, it should be noted that some lists focused on the minimum physicochemical parameters which should be reported also suggest certain experimental conditions such as "particle concentration" 3 and "media" 60 should be reported. (Here, the potential ambiguity as to what is considered a physicochemical parameter for a nanomaterial sample and what is considered an experimental condition should be noted: "particle concentration" 3 and "pH" 77 may be considered either as physicochemical properties or important experimental conditions.) 36 Other proposals, such as the caNanoLab data availability standard, 78 go further and stipulate that other (meta)data, such as characterisation with respect to specific biological endpoints, should be made available.
Key international standards bodies, the Organisation for Economic Co-operation and Development (OECD) and the International Standards Organisation (ISO), have also made recommendations regarding physicochemical parameters and other experimental variables which should be reported for various kinds of experimental studies of nanomaterials. [79][80][81][82][83][84][85] Notable reports include the "Guidance Manual for the Testing of Manufactured Nanomaterials: OECD Sponsorship Programme" 80 which stipulates physicochemical parameters and biological endpoints which needed to be assessed, as part of the OECD's "Safety Testing of a Representative Set of Manufactured Nanomaterials" project, and a guidance document on sample preparation and dosimetry, 81 which highlights specific experimental conditions, associated with stepwise sample preparation for various kinds of studies, that should be reported.
Many of the proposals cited above are not associated with a specific curated nanomaterial data resource, although some which were intended as recommendations for experimentalists (e.g. the MINChar Initiative Parameters List) 60 have been used as the basis for curated data scoring schemes. 78 Examples of proposals which are specifically used as the basis of a scoring scheme, partly or wholly based upon data completeness, for curated nanomaterial data include those employed by the Nanomaterial Registry, 39,86,87 caNanoLab 78 as well as the MOD-ENP-TOX and ModNanoTox projects (see ESI †).
Some proposals draw a distinction between broader completeness criteria (see Table 1) and what may be considered "minimum information" criteria (see Table 2). For example, within the MOD-ENP-TOX project (see ESI †) a set of minimum physicochemical parameters were required to be reported within a publication in order for it to be curated: composition, shape, crystallinity and primary size. Additional physico-chemical parameters (such as surface area) were deemed important for the data to be considered complete. This is in keeping with many proposals reviewed by Stefaniak et al., 3 which drew a distinction between "properties of interest" and "minimum" (or "priority") properties, as well as publications proposing increasing characterisation requirements within a tiered approach to nanosafety assessment. 67,68 Some proposals have also stressed the context dependence of completeness definitions. For example, the ModNanoTox project proposed (see ESI †) that certain physicochemical parameters and experimental metadata were only relevant for certain kinds of nanomaterials: crystal phase was considered crucial for TiO 2 nanoparticles but less important for CeO 2 nanoparticles, in keeping with an independent review of the literature emphasising the importance of crystal phase data for TiO 2 nanomaterials specifically. 68 Recent publications have also stressed the importance of characterisation requirements depending upon the type of nanomaterials studied and otherwise being relevant for the specific study. 68,88,89 Indeed, in contrast to the proposals discussed above which define specific (meta)data requirements, the developers of the Center for the Environmental Implications of NanoTechnology (CEINT) NanoInformatics Knowledge Commons (CEINT NIKC) data resource [90][91][92] have proposed that data completeness be calculated on a use-case-specific basis i.e. with respect to the (meta)data which a given database query aims to retrieve. For example, a researcher interested in the die off rate of fish due to nanomaterial exposure would need mortality data at multiple time points, whereas a researcher interested in mortality after, say, one week would only need data at a single time point.

An overview of nanomaterial data quality assessment proposals
Various schemes for scoring/categorising nanomaterial data (in part) according to their quality have been proposed in recent years. Because data completeness (see Table 1) and quality (see Table 3) may be considered highly interrelated, a number of these schemes are strongly based upon consideration of (meta)data availability. One of the simplest schemes, presented by Hristozov et al., 93 assessed the reliability of toxicity data in nanomaterial databases based purely upon the availability of basic provenance metadata: data were considered "unusable", or "unreliable", where a result from a study is not accompanied by a "properly cited reference". Significantly more sophisticated schemes exist which take into account the availability of a variety of additional (meta)data such as the availability of certain physicochemical data and experimental details concerning biological assay protocols. One such sophisticated scheme is the iteratively developed DaNa "Literature Criteria Checklist" 75,76 used to assess the quality of a given published study concerning a given nanomaterial for the purpose of preventing low quality scientific findings from being integrated within the DaNa knowledge base. [94][95][96] Indeed, some existing nanomaterial quality proposals go beyond merely considering data completeness, but are also concerned with whether the experimental protocols were carried out appropriately. For example, Lubinski et al. 47 proposed an extension of the Klimisch framework 48 for evaluating the reliability of nanotoxicology, or nano-physicochemical, data which was considered, in part, to depend upon compliance with Good Laboratory Practice (GLP) 97 and standardised test protocols. Other assessment schemes, such as the scheme employed by the DaNa 75,76,94-96 project (see ESI †), take account of whether biological results were affected by assay interference. [98][99][100][101][102][103][104][105][106][107] Indeed, application of the DaNa "Literature Criteria Checklist" 75,76 entails making a range of judgements regarding the quality of the nanomaterial data which go beyond mere consideration of data completeness (see ESI †). Likewise, Simkó et al. proposed a range of criteria for evaluating in vitro studies, including clearly specified criteria for the statistical "quality of study". 108 Some, but not all, proposals for quality assessment of nanomaterial data have sought to assign a categorical or numeric score to express the quality of the nanomaterial data. One such scheme, which assigns a qualitative score, was proposed by Lubinski et al. 47 Likewise, the "Data Readiness Levels" scheme proposed by the Nanotechnology Knowledge Infrastructure (NKI) Signature Initiative 51 assigns any kind of datai.e. not necessarily generated for nanomaterialsto one of seven, ranked categories denoting their "quality and maturity". In contrast, the following schemes assign numeric quality scores and were specifically designed to evaluate nanomaterial data curated into a specific data resource. The Nanomaterial Registry, 109,110 assigns normalised, numeric "compliance" scores to each nanomaterial record in the database based upon its associated measurements, corresponding to the physicochemical characteristics specified in the "minimal information about nanomaterials (MIAN)", which are designed to capture the "quality and quantity" of the physicochemical characterisation performed for that nanomaterial. 39,86,87 The MOD-ENP-TOX and ModNanoTox curated nanomaterial data resources also developed quality scoring schemes which assign numeric scores (see ESI †).
One notion of data quality (see Table 3) might be based on validation of dataset files, according to their data content or compliance with format specifications, using specialist software tools. (This is further discussed in section 4, with examples from mature fields.) In the nanoscience area, the validation tools 111 developed within the MODERN E.U. FP7 project, 112 used to validate ISA-TAB-Nano datasets based on their compliance with the ISA-TAB-Nano specification, [113][114][115] were, to the best of the authors' knowledge, the only such tools available at the time of writing which were specifically developed for validating curated nanomaterial datasets.

Lessons which can be learned from mature fields
In order to improve the means via which the completeness and quality of (curated) nanomaterial data are currently evalu-ated, it is worth considering the lessons which may be learned from "mature" fields.
A variety of different minimum information checklists or reporting guidelines (see Table 2) have been proposed in different areas of the life sciences. These are increasingly being used by publishers to assess the suitability of submitted publications. [116][117][118] The seminal Minimum Information About a Microarray Experiment (MIAME) reporting guidelines were proposed over a decade ago to describe the minimum information required for microarray data to be readily interpreted and for results obtained from analysis of these data to be independently verified, 116,119 which may be achieved if the results are reproducible. In under a decade, this standard was widely accepted and most scientific journals adopted these guidelines as a requirement for publication of research in this area, with authors being obliged to deposit the corresponding MIAMEcompliant microarray data in recognised public repositories. 116 A variety of similar guidelines 116 were subsequently developed for other life science technologies (e.g. proteomics) 120 or studies (e.g. toxicology 121 and molecular bioactivity studies). 122 The BioSharing project and online resource, [123][124][125][126] originally founded as the MIBBI Portal in 2007, 28 serves to summarise proposed "reporting guideline" standards and promote their development and acceptance. Clearly, the BioSharing online resource might be used to link to the various minimum information checklists that have been (implicitly) developed within the nanoscience domain (see section 3.1), thereby raising awareness of them and facilitating their comparison and further development. It is also possible that some of the recommendations made regarding experimental (meta)data in the (non-nanoscience specific) reporting guidelines linked to via the BioSharing website may also be applicable to (specific sub-domains of ) the nanoscience area.
The Standard Reference Data Program of the U.S. National Institute of Standards and Technology (NIST) 127 has supported the evaluation of data in many areas of science and technology. Typically, data are not only curated but also evaluated from three perspectives: documentation of the identification and control of the independent variables governing a measurement; the consistency of measurement results with the laws of nature; and through comparison with similar measurements. Over the years it has become clear that, as new phenomena are identified and measured, it takes yearsif not decadesto truly identify and understand how to control a measurement. Consequently, initial experiments produce data that primarily provide guidance for future experiments rather than be recognised as definitive properties. Feedback from the evaluation efforts to the experimental community is critical for improving the quality of data.
Chirico et al. 53 recently described how NIST data resources and computational tools can be and are being used to improve the quality of thermophysical and thermochemical data submitted for publication within the context of a collaborative effort between NIST and five key journals.
Because uncertainty may be considered a key aspect (Table 3), or even the key aspect, 25,52 of data quality evaluation, the approaches to characterising uncertainty proposed by ISO, 25,52 NIST 32 and SCENIHR 23 merit consideration.
The concept of data quality has received considerable attention within the toxicology and risk assessment communities and a number of proposals for assessing the quality of data, studies or publications have been published. 23,48,[128][129][130][131][132] A number of these were reviewed in Ågerstrand et al. 133 and Przybylak et al. 49 Arguably the most well-known is the framework proposed by Klimisch et al. 48 for categorising the reliability (see ESI Table S5 † literature definition 3.4) of toxicology data, or a toxicology study test report or publication. The Klimisch categories are widely employed within regulatory toxicology. 24,49,132,134 Since the original work of Klimisch et al. 48 lacked detailed criteria for assigning their proposed reliability categories, the ToxRTool program 131,135 was proposed as a means of improving the transparency and consistency with which these categories were assigned. The program assigns a reliability category based upon the score obtained after answering a set of "yes/no" questions. However, it is interesting to note that neither GLP nor test guideline compliance is explicitly considered by the ToxRTool when assessing reliability (although these issues are considered when evaluating "relevance")even though these were deemed key indicators of reliable data in the original work of Klimisch et al. 48 Recently, an extension to the ToxRTool program was developed by Yang and coworkers. 136 Their approach took the following issues into account: (1) an assessor might feel that a given ToxRTool criterion was only partially met, rather than it being possible to simply answer "yes/no" for that question; (2) an assessor might be unsure of the most appropriate answer to a given question. Hence, their approach, based on fuzzy arithmetic, allows toxicity data to be assigned to multiple reliability categories with different degrees of satisfaction.
Consideration of these different approaches to evaluating data quality raises some important questions which arguably need to be taken into account when designing a scheme for assessing the quality of nanosafety data or, where applicable, nanoscience data in general.
1. To what extent should quality be assessed on the basis of considering data completeness as opposed to making judgements regarding the data such as the "soundness and appropriateness of the methodology used" 23 or, equivalently, whether or not a method was "acceptable"? 48 2. More specifically, should data be considered most reliable 48 when they were generated according to Good Laboratory Practice (GLP), 97 or some other "audited scheme" 23 and according to standardised test protocols, 133 such as those presented in OECD Test Guidelines or by ISO? The appropriateness of adherence to standardised test protocols is especially relevant for testing of nanomaterials (see section 5.11). It may also be argued that, even for conventional chemicals, data which were not generated according to standardised test protocols and/or GLP are not necessarily less reliable. 48,132,137 3. To what extent should a data quality assessment scheme be prescriptive as opposed to allowing for flexibility based upon expert judgement? Whilst a scheme which is more prescriptive offers the advantage of promoting transparency and consistency 23,131 in the assigned quality scores (or categories), flexibility based upon allowing for expert judgement may still be necessary. 23 4. Should the outcome of the quality assessment be expressed numerically? Beronius et al. 132 have argued that this risks implying an undue level of scientific certainty in the final quality assessment. However, using a qualitative scheme based on certain criteria being met in order for data to be assigned to a particular category would fail to assign partial credit to data meeting a subset of those criteria. Furthermore, as illustrated by the ToxRTool approach, 131,135 a numeric score might be mapped onto a qualitative category for ease of interpretation.
5. How can the community best characterise uncertainty to provide a clearer understanding of data quality?
The preceding discussion concerns proposals which might be applied by a human expert for the purposes of assessing data completeness and quality in various domains. In principle, where these schemes are sufficiently prescriptive, rather than relying on subjective expert judgement they could be applied programmatically i.e. via parsing a structured electronic dataset or database using specialist software.
Indeed, various validation software programs have been developed to validate electronic datasets, based on standardised file formats, according to a range of criteria. For example, validation programs have been developed to validate different kinds of biological (meta)data reported in XMLbased 58,59,138 or ISA-TAB 139,140 formats and, more specifically, raw sequence and sequence alignment data [141][142][143][144] reported in FastQ [142][143][144] or Binary Alignment/Map (BAM) format. 145 Validation software 146,147 was also developed for crystallographic data reported in the crystallographic information file (CIF) format. 148 As well as checking format compliance, some of these validation programs may also be used to enforce compliance with (implicit) minimum information checklists. 138,149 For example, The Cancer Genome Atlas (TCGA) 150 validation software checks certain fields to ensure they are not "null" (unknown) or missing, as well as carrying out various other data quality checks for errors and inconsistencies. 138 Software used to validate sequence data may carry out data quality assessment via calculating a variety of metrics, including those which are indicative of different kinds of possible errors/biases/artefacts generated during measurement/analysis or possible contamination of the analysed samples. [142][143][144] All of these software programs are potentially relevant to automatically validating nanomaterial characterisation and/or biological data. The ISA-TAB format [151][152][153] was recently extended via the development of ISA-TAB-Nano [113][114][115] to better capture nanomaterial (meta)data, so the ISA-Tools 139,140 software might be extended to validate ISA-TAB-Nano datasets. (As is discussed in section 3.2, some software for validating ISA-TAB-Nano files already exists.) 111,115 Validation software for CIF files is arguably of particular relevance to building quanti-tative structure-activity relationships (QSARs), or quantitative structure-property relationships (QSPRs), for nanomaterials. Crystallographic data has been used to calculate descriptors for nano-QSAR (or nano-QSPR) models of inorganic oxide nanoparticle activities (or properties) in various recent studies. 42,154,155 5. Key challenges Important challenges are associated with nanomaterial data which need to be taken into account when evaluating their completeness and quality. To some extent, a number of these issues are taken into account in a subset of the existing proposals for evaluating nanomaterial data (see section 3). Other challenges relate to limitations of (some) of these existing evaluation proposals. The key challenges are summarised in Table 4 and explained in the remainder of section 5.

Uncertainty regarding the most biologically significant variables
A key challenge associated with defining minimum information criteria for nanomaterials is that the current understanding of the independent variables, such as nanomaterial physicochemical properties and other experimental variables, which contribute most significantly to the variability in the outputs of biological assays is arguably insufficient. 3,41,[68][69][70]89,105,156 Understanding which of the physicochemical properties are most correlated to biological effects is hampered by the dependence of many of these properties on experimental conditions (section 5.2), time (section 5.3), dosimetry uncertainty (section 5.4), possible redundancy in physicochemical data (section 5.5), the potential for artefacts in biological studies related to the presence of nanomaterials (section 5.9) and possible confounding factors (section 5.10). Table 4 The key challenges which impact completeness and quality evaluations of (curated) nanomaterial data Challenge no.
Brief description 5.1 Uncertainty regarding the most biologically significant variables 5.2 Dependence of many physicochemical properties on experimental conditions 5.3 Potential time dependence of physicochemical properties 5.4 Problems expressing dosimetry in biological assays 5.5 Possible redundancy in physicochemical data 5. 6 Batch-to-batch variability of nanomaterials 5.7 Context dependency of (meta)data requirements 5. 8 Lack of clarity in some existing checklists 5. 9 Artefacts in biological studies related to nanomaterials 5. 10 Misinterpretations in biological studies 5. 11 Uncertainty regarding standardised test guidelines 5. 12 Reduced relevance of some standard assays 5. 13 Problems with analysis of environmental samples

Dependence of many physicochemical properties on experimental conditions
Many, but not necessarily all, physicochemical parameters may change significantly depending upon the dispersion (suspension) medium and any additives (e.g. dispersant aids) 37 i.e. many physicochemical characterisation data obtained under pristine conditions (e.g. dispersed in water) may differ greatly from those determined for the nanomaterial dispersed in the medium, plus additives, used for biological testing. [36][37][38][39][40]157 This variability makes it difficult to find correlations between the physicochemical properties and the outcome of biological assays. No straightforward relationship can be expected to exist when these properties are measured under pristine conditions, or conditions which otherwise differ from biologically relevant conditions, even if a simple correlation exists when the physicochemical properties are measured under biologically relevant conditions. For example, a recent study found the positive zeta potential values measured in physiological saline ( pH 5.6) exhibited good linear correlation with acute lung inflammogenicity, but not the negative values measured in more basic ( pH 7.4) media. 157 Other experimental conditions which may significantly affect physicochemical properties include sample processing details such as sonication steps. 37 As well as making it harder to discern which physicochemical parameters are most important to measure and document, this challenge has the following implications for data completeness. Firstly, a careful description of the various factors which could affect physicochemical properties is required 36,38,40,81 in order to establish "uniqueness" and "equivalency" 41 based upon physicochemical characterisation. Secondly, measurement of many physicochemical characteristics under biologically relevant conditions, as is considered best practice, 38 should assist with explaining biological results or developing structure-activity relationships.

Potential time dependence of physicochemical properties
Many nanomaterial characteristics may change over time, depending upon their environment and processing protocols, such as their state of agglomeration, 40,81 their "corona" 158-160 of adsorbed (biological) molecules 40,161 and even primary particle characteristics such as chemical composition (e.g. via dynamic speciation) 162,163 or morphology. 37 Some of these changes may be reversible, 159,164 whilst other processes may give rise to irreversible transformations or "aging" 165 ("ageing"). 166 These time dependent changes in physicochemical properties can give rise to changes in their biological effects. 166 The first implication for data completeness is that temporal metadata, 166 along with corresponding processing (e.g. sonication) 37 and storage history 166 details, are important to capture. Secondly, because "ageing" may have transformed the physicochemical characteristics responsible for biological activity, data for biological studies of nanomaterials might not be considered complete if key physicochemical characteristics were not measured at time points corresponding to biological testing. 166

Problems expressing dosimetry in biological assays
The most appropriate dose metric to use in biological studies of nanomaterials is unclear and may depend upon the kind of nanomaterial being considered. 167 Nonetheless, it is generally accepted 77,81,167,168 that mass based concentrations and doses are less appropriate and that dose metrics based on the total surface area or number of particles should be considered: the use of mass based concentration units may give misleading indications as to the rank order of toxicity for different nanomaterials. 77 Thus, the use of an inappropriate dose (or concentration) metric may be considered to adversely affect the clarity, hence the quality (see Table 3), of nanomaterial biological data. Since additional physicochemical data are required for conversion of the mass based concentration (or dose) units (e.g. surface area measurements or density measurements, depending upon the approach employed), 36,77,81,168 this issue also has implications for the minimum information criteria which might be proposed for nanomaterial data. N.B. Different approaches for estimating surface area based dose units, based upon different physicochemical measurements, have distinct advantages and disadvantages: geometric estimates of surface area may be based upon simplistic assumptions regarding particle geometry and fail to take account of porosity, whilst surface area measurements under dry conditions may not reflect the accessible surface area under biological conditions. 36,168 An additional problem is that the nominal, administered concentration (or dose) may not correspond to the concentration (or dose) delivered to the site of biological action. 101,[168][169][170] Hence, additional data completeness considerations for aquatic toxicity tests include measurements of exposure levels over the course of the experiment and data quality concerns arise regarding whether the experimental methods employed to quantify nanomaterials in complex media are appropriate (see section 5.13). 101,169

Possible redundancy in physicochemical data
As discussed in section 5.4, different kinds of physicochemical data may be required to estimate surface area based dose units, depending upon the approach employed i.e. this is one source of potential redundancy in physicochemical characterisation requirements. However, as is also discussed in section 5.4, even ignoring other rationales for obtaining the same physicochemical data, the different strengths and weaknesses of alternative surface area based dosimetry approaches mean these data cannot be said to be completely interchangeable. The interrelatedness between nanomaterial physicochemical properties 44,68,154 also means that, in principle, extensive lists of "essential" properties 3 may call for excessive characterisation that is both a burden for experimentalists and curators. However, the degree of interrelatedness between physicochemical properties may not mean that some properties are entirely interchangeable and, furthermore, the relationships between different propertiesespecially if measured under different conditionsare arguably hard to discern. 68 Indeed, investigating which properties correlate might be hampered by synthesis challenges 5 which may be associated with producing systematically varied nanomaterial libraries. 171 Given the lack of complete interchangeability and problems associated with determining correlations in physicochemical properties, reducing the necessary physicochemical characterisation data based on potential redundancy remains a challenge. Furthermore, a challenge which arises as a consequence of these correlations is that it may be difficult to interpret the effect of changing a given property upon biological activity (hence, the importance of measuring that property) without this being confounded by variation in other physicochemical parameters.

Batch-to-batch variability of nanomaterials
The issue of batch-to-batch variability, i.e. variability in the properties of nominally identical nanomaterials obtained via repetitions of nominally the same synthesis, is a key challenge which is particularly significant for industrially produced nanomaterials. 5,38,172 The implications for data completeness are arguably that the batch identity of a given nanomaterial (as denoted via its "batch identifier", 38,173 "lot number" 38,173 or "manufacturer lot identifier") 174 should be documented, to establish nanomaterial "equivalency", 41 even for nanomaterials which are nominally the same e.g. which have the same trade name. However, since not all nanomaterial synthesis procedures may exhibit the same degree of batch-tobatch variability, 5,38,172 the importance of these metadata may depend upon the kind of synthesis procedure. Indeed, the kind of synthesis route may be considered important metadata to curate 174 for this reason and because it may implicitly convey (biologically relevant) information regarding chemical composition. 175

Context dependency of (meta)data requirements
Data and metadata requirements may depend upon the experimental scenario and intended use of the data i.e. the specific context. Not all (meta)data are relevant for all experimental scenarios. For example, not all physicochemical parameters are applicable to all kinds of nanomaterials and those physicochemical parameters which contribute most significantly to nanomaterial effects may vary according to the kind of nanomaterial, their intended application and the specific effect of interest. 3,68,69,83,88 Likewise, not all of the key experimental variables which (most) affect the outcome of biological testing will necessarily be common to all kinds of biological assays. 105 For example, whether cytochalasin-B is employed during a micronucleus assay, which may be used to evaluate the genotoxicity of nanomaterials, 6,176 can significantly affect the results. 176,177 However, this experimental variable is not relevant for other genotoxicity tests. 6,176 Moreover, in practice, different stakeholders will have different objectives i.e. the pro-perties and experimental metadata which are important may vary between disciplines and user communities, or even within the same disciplines and communities the information requirements may vary according to the specific questions posed of the data. 41 Hence, enforcing a single set of "minimum information" criteria could lead to some existing data being unnecessarily deprecated due to a lack of completeness even though the existing (meta)data are sufficient for specific purposes. 89 For example, consider toxicological assessment of a commercially available nanomaterial with limited batch-to-batch variability, 5,38,172 assessed during different studies at essentially the same point in its life-cycle or which is not significantly affected by "aging". 165 For such a nanomaterial, its trade name ("X") might be considered a sufficiently unique identifier i.e. one can suppose that essentially the same material is being referred to in different studies of "X" or that the samples being assessed do not cause significantly different biological effects for the endpoint(s) of interest. If these data were simply being used to determine whether material "X" could cause a given set of effects (as determined in different studies), enforcing a requirement for adherence to a "minimum information checklist" in terms of physicochemical characterisation 3 might be considered unnecessarily stringent i.e. in this context, detailed physicochemical characterisation might not be required to establish "equivalency". 41 Conversely, if a nano-QSAR modeller wished to generalise from these data (e.g. to build a relationship between physicochemical characteristics and a given adverse effect), then batch-specific physicochemical characterisation might be considered much more important.
In light of the context dependence discussed here and the evolving state of nanoscience (e.g. challenge 5.1), those utilising stringent "minimum information" schemes should anticipate that their criteria are not necessarily applicable in all contexts and are likely to be superseded as the field develops, instruments improve, and current hypotheses are exhausted. However, the underlying informational value of current and past data may nevertheless remain intact.

Lack of clarity in some existing checklists
Many existing proposals regarding important physicochemical data specify characteristics which are very broadly defined, rather than a specific set of measurements, 3 making it unclear to researchers as to which measurements should be made. For example, many lists propose that the "agglomeration" or "aggregation" state be determined. 3 However, a variety of different measurements (such as number of primary particles per aggregate or agglomerate, as might be quantified via the "average agglomeration number", 178 or assessment of particle size distributions under different conditions) might be considered to assess this. 36,179 A related issue is that two protocols which are nominally measuring the same parameter (such as "average size"), may actually be providing different kinds of information that are not directly comparable. 3,36,38,180 Different measurement techniques, such as transmission electron microscopy (TEM) and dynamic light scattering (DLS), employ different principles and assumptions to estimate "size" and may be measuring different aspects of "size" (e.g. "height above a flat substrate" or "hydrodynamic diameter"). 3,180,181 Some techniques (e.g. TEM) may be used to estimate the "size" of agglomerates, aggregates or the primary particles, depending upon how the raw data are analysed, 37,182 and different kinds of "average size" may be obtained using the same technique. 36,180,181 The implications for data completeness are that (1) recommendations for specific kinds of physicochemical data, or clear guidance regarding acceptable alternatives, should be provided and (2) corresponding metadata regarding the measurement technique, the characterisation protocol and a precise description of the kind of statistical estimate produced (e.g. arithmetic mean of the number distribution vs. volume distribution) 36 are important to capture.

Artefacts in biological studies related to nanomaterials
A growing body of literature has raised concerns regarding various artefacts which may affect the reliability of biological assessment of nanomaterials. 70,98,99,[101][102][103][104][105][106][107][183][184][185][186] These artefacts mean that the measurements obtained may not entirely correspond to the biological phenomena which the studies are trying to detect. For example, various kinds of nanomaterial "interference" with commonly used in vitro (cell-based) toxicity assays have been noted which may lead to overestimation or underestimation of toxicity. 70,98,99,[102][103][104][105][106][107][183][184][185][186] In in vivo aquatic toxicity studies, nanomaterials adhering to the surface of organisms may inhibit movementleading to overestimation of mortality. 101,187 An immediate implication for evaluating the quality of (curated) nanomaterial data is the need to evaluate the possibility for artefacts (e.g. interference). This is complicated by the fact that assay interference may be dependent upon the specific combination of assay, nanomaterial and tested concentration. 98,106,185,186 Indeed, the possible dependency of assay interference on specific physicochemical characteristics 106,185 may be another factor to take into account when evaluating completeness and quality.
Various recommendations have been made in the experimental literature for detecting and, in some cases, correcting for possible assay interference. [104][105][106]184,185 In spite of this, analysis by Ong et al., 185 using a sample size of 200 papers for each year, suggested that around 95% (90%) of investigations using colorimetric or fluorescent-based assays published in 2010 (2012) failed to experimentally assess the potential for nanomaterial interference or, at least, failed to explicitly state that such potential interferences had been ruled out experimentally.

Misinterpretations in biological studies
As well as artefacts which lead to erroneous estimations of toxicity, a variety of additional factors may lead to erroneous interpretation of the cause of the toxicity observed when testing nanomaterials. 104 For example, a failure to experi-mentally determine the presence of different kinds of impurities (e.g. endotoxin contamination, solvent contamination, metals) may lead to the observed toxicity being wrongly attributed to the nominal nanomaterial. 104 The implications for data completeness are that thorough characterisation of the nanomaterial, including with respect to these key impurities, needs to be carried out when studying the biological effects of nanomaterials in order to meet the following objectives: (1) unless the nanomaterial identity is otherwise clear (see section 5.7), to associate a specific nanomaterial identity with the observed biological activity; (2) if desired, to ensure that any mechanistic interpretation of the biological effect is correct. Lack of clarity in the meaning of the data, such as failure to correctly identify which specific nanomaterial was tested in an assay, can also be considered to affect data quality (see Table 3).

Uncertainty regarding standardised test guidelines
An initial review 188 of the applicability of the OECD test guidelines to nanomaterialsdeveloped as standardised test protocols for conventional, small molecule chemicals 101,188concluded that many (but not all) of these were applicable to nanomaterials in principle, if coupled with additional guidance documents regarding nanospecific issues. 81,188 A related question is the requirement for OECD test guidelines for parameters which are specifically important for nanomaterials. 68,189 However, these issues were still not fully resolved as of the time of writing. 68,79,169,179,[189][190][191] Also, at the time of writing, some standardised protocols for nanomaterial assessment with respect to a variety of endpoints were under development by ISO. 82,192 Nonetheless, some recent articles in the nanotoxicology literature have strongly advocated the use of OECD test guidelines, or other standardised protocols, to evaluate nanomaterials. 103,193,194 Clearly, if the use of established standardised protocols cannot be assured to address all of the concerns raised regarding the quality of nanomaterial data (e.g. the artefacts discussed in section 5.9), this has implications as to whether adherence to existing standardised protocols should be considered an indicator of high quality data, as supposed by some existing data quality evaluation schemes discussed in sections 3 and 4, 47,48 compared to a novel protocol which may have been specifically designed to address these concerns. Indeed, it is, in principle, possible that the use of some existing standardised tests might miss novel endpoints or be based upon assumptions regarding the mode of action that are not applicable to some nanomaterials. For example, the use of "omics" methods in nanotoxicology is advocated due their ability to capture novel modes of action. 195 However, the extent, if any, to which nanomaterials can cause novel harm, act via genuinely novel modes of actionor even exhibit novelty in the underlying 45 mechanisms of action and/or structure activity relationshipshas recently been debated. 14,89,167,[196][197][198][199] 5.12. Reduced relevance of some standard assays Another potential problem with some toxicity tests when applied to nanomaterials, as compared to testing of small molecules, is that they might be of reduced relevance for assessment of possible human health effects. For example, the Ames genotoxicity test and cytotoxicity tests, based on bacterial cell cultures, might be inappropriate for nanomaterials as bacterial sensitivity to nanomaterials may be significantly reduced compared to human cells, 176,200 due to reduced uptake as a result of the cell wall and lack of endocytosis for bacterial cells. 176 However, it should be noted that Holden et al. 201 have suggested that bacterial studies may still be relevant to assessing potential nanomaterial impacts on human health, at least in terms of indirect effects following environmental release.
Reduced relevance for human health effects assessment is sometimes considered to be a data quality issue (see ESI  Table S5 † literature definition 3.4). 48

Problems with analysis of environmental samples
The analysis of engineered nanomaterials, along with their derivatives, in environmental samples provides important information for risk assessment. 202 The engineered nanomaterials first need to be detected, followed by quantification of their concentration and determination of their physicochemical properties. 202 In particular, quantification of their concentration provides a direct means of validating the predictions of fate and transport models. 203 However, obtaining reliable data on engineered nanomaterials in environmental samples remains challenging. 202,203 In part, this reflects the need to make measurements at or below the detection limits for many analytical techniques. For example, many analytical techniques (e.g. dynamic light scattering) have detection limits 101,[203][204][205][206] which are too high to detect concentrations as low as those expected for engineered nanomaterials in environmental samples. 101,203,205 Recently, single particle inductively coupled plasma mass spectrometry (SP-ICP-MS) has been advocated as a possible solution which would allow detection of realistic environmental concentrations and, in combination with additional information or assumptions, simultaneous measurement of particle size distributions. 202,203,205 However, SP-ICP-MS is not without its limitations, 202,203 including composition dependent size detection limits. 205 (Indeed, detection of small particles is noted to be a problem with many analytical techniques due to their detection limits and/or low sensitivity for smaller particles.) 206,207 In addition to these challenges, it has been argued that the most serious remaining problem with analysis of engineered nanomaterials in environmental samples is discriminating engineered from naturally occurring nanomaterials. 203 The key challenges highlighted in this section emphasise the difficulties associated with generating sufficiently complete and high quality nanomaterial data. Consideration of these challenges is critical when evaluating the completeness and quality of (curated) nanomaterial data.

Recommendations for promoting and improving upon established best practice
The following recommendations are designed to promote established best practice or improve the manner in which the completeness and quality of curated nanomaterial data are evaluated. Many of these recommendations are also applicable to evaluating the completeness and quality of nanomaterial data reported in, say, the published literature prior to curation. They were informed by the preceding discussions regarding the meaning and importance of data completeness and quality (section 2), existing proposals for evaluating the completeness and quality of (curated) nanomaterial data (section 3), lessons which can be learned from mature fields (section 4) and the key challenges associated with nanomaterial data (section 5). These recommendations were developed by the authors of the current publication and were informed by the responses to the Nanomaterial Data Curation Initiative (NDCI) survey on data completeness and quality. (Full details of the recommendations made by specific survey respondents may be found in the ESI. †) However, they should not be considered to provide a definitive road-map for progress in this area which is endorsed by all authors and survey respondents. Rather, they summarise options for promoting best practice or improving the evaluation of the completeness and quality of curated nanomaterial data.
These recommendations are divided into five categories: terminology recommendations (section 6.1), specific (meta) data requirements (section 6.2), computational tool focused recommendations (section 6.3), strategic recommendations (section 6.4), and recommendations regarding the role specific organisations and scientific communities could play in advancing the manner in which the completeness and quality of curated nanomaterial data are evaluated (section 6.5).
To allow the reader to get a quick overview, the recommendations are merely summarised in the main text of the article. An in-depth discussion of these recommendations, including caveats, is provided in section S4 of the ESI. †

Terminology recommendations
It is proposed that the following definitions of terms (Table 5) should be adopted across the nanoscience community. The particular context in which these terms are explained is nanomaterial data curation. However, the definitions and many of the accompanying notes are relevant to the wider nanoscience, or broader scientific, community. These definitions build upon the broad and flexible definitions of (curated) data completeness (Table 1) and quality (Table 3) presented in section 2.
The new definitions are generally consistent with the definitions presented in section 2. However, some issues incorporated into those broad and flexible definitions are deemed out of scope. For example, it is proposed that the relevance of the data for a particular purpose should be considered related to data completeness rather than quality. The broad and flexible definitions (section 2) were appropriate for reviewing prior work as they ensured that different perspectives were not deemed out of scope. However, for the sake of greater clarity, the following, specific definitions are recommended to the community. This greater clarity will aid consideration of the practical recommendations presented in the remainder of this article.
6.1.1. Specific definitions of completeness and quality are recommended to the nanoscience community. The terms data completeness and quality should be considered to be related but should not be used interchangeably. Guidance notes which further clarify the following definitions are presented in the detailed discussion of these terminology recommendations in ESI S4. † Data completeness. This is a measure of the extent to which the data and metadata which serve to address a specific need are, in principle, available.
Data quality. This is a measure of the degree to which a single datum or finding is clear and the extent to which it, and its associated metadata, can be considered correct.
These abstract definitions are further clarified by Fig. 1, which illustrates the kinds of (meta)data requirements for data to be assessed as sufficiently complete and of acceptable quality. A more detailed discussion of specific (meta)data requirements is provided in section 6.2.
6.2. Specific (meta)data requirements 6.2.1. Specific (meta)data highlighted by the NDCI survey. The Nanomaterial Data Curation Initiative (NDCI) survey on data completeness and quality asked respondents to suggest the different kinds of (meta)data required in order for nanomaterial data to be considered sufficiently complete and of sufficient quality.
They were further asked to consider whether these (meta) data were only important in specific contexts and to identify those (meta)data they felt were most important to capture. The aim here was to capture recommendations even if they went beyond the (meta)data considered when curating the nanomaterial data resource for which they were acting as a liaison. (See the ESI † for further details.) Some survey respondents emphasised that their responses were not intended to be a comprehensive summary of all (meta)data and considerations which would need to be taken into account in order to assess the completeness and quality of curated nanomaterial data. Rather, their responses to these questions highlighted issues (e.g. nanomaterial ageing) which they considered to be given insufficient attention. Some respondents kindly provided detailed lists of (meta)data and comments regarding additional considerations required for completeness and quality assessment. Some of these responses also gave some consideration to the relative importance and context/use-case dependence of certain kinds of (meta)data requirements.
The recommendations regarding physicochemical data which should be provided were generally in keeping with the kinds of physicochemical data recommended as being important in the lists analysed by Stefaniak et al. 3 As well as physicochemical data, many kinds of metadata were also highlighted as being important for data to be determined to be sufficiently complete and/or of sufficient quality. Metadata recommendations were concerned with various issues, including experimental conditions, protocols and techniques, as well as data provenance, nanomaterial synthesis and experimental error.
Based on the survey responses and the literature review which informed the current article, a definitive list of all necessary (meta)data cannot be made. Neither can a definitive set of lists presenting all (meta)data requirements for different scenarios be made. Nonetheless, some key recommendations may be made.
6.2.2. Key recommendations regarding specific (meta)data. Table 6 presents key recommendations concerning specific kinds of (meta)data which are important to capture in various curated nanomaterial data collections. ESI S4 † explains these recommendations in detail.
It should be noted that these recommendations are not a comprehensive list of all kinds of (meta)data which need to be captured in curated nanomaterial data collections. Rather, they are designed to emphasise key issues which are not always captured in existing minimum information checklists (section 3.1) or quality assessment schemes (section 3.2) for (curated) nanomaterial data. Additional (meta)data requirements might be determined via consulting existing proposals (see sections 3.1 and 3.2). Indeed, the need to consult existing recommendations is a key strategic recommendation (recommendation 6.4.1).
However, the possible dependence of (meta)data requirements upon the kinds of data and intended use of those data must be remembered (see section 5.7). This consideration is applicable, in principle, to the existing proposals (see sections 3.1 and 3.2) as well as the recommendations in Table 6. To some extent, the context dependence of the recommendations is indicated in Table 6. The discussion of these recommendations in ESI S4 † considers this context dependence in greater depth. Table 7 presents recommendations regarding how computational tools might be developed to support evaluation of the completeness and quality of curated nanomaterial data. Some of these recommendations concern existing nanoinformatics Fig. 1 The quality and completeness of (curated) nanomaterial data are viewed as overlapping, yet distinct, concepts. This figure illustrates various contexts, meaning the experimental scenario and intended use of the data, and the kinds of (meta)data which may be required to assess those data as being sufficiently complete and of acceptable quality. N.B. (1) PCCs is an abbreviation for physicochemical characteristics. (2) The concept of data completeness applies to a set of data and their associated metadata. Hence, the number of data points of a specific kind (e.g. number of nanomaterials screened in a cytotoxicity assay) may be a completeness criterion in specific contexts if a given number of data points are required to achieve a specific aim. (3) In contrast, the concept of data quality applies to a single datum (i.e. a single data point) or a single "finding", taking into account its associated metadata. A "finding" might be a conclusion derived from analysis of a set of raw or processed data and the "metadata" associated with that finding might include these data. (4) The dependence of both completeness and quality upon metadata is not entirely for the same reasons. For example, metadata (e.g. related to the nanomaterial identity and experimental conditions) are required to determine the relevance of the data for answering a specific question. The relevance of data for answering a specific question affects the completeness of the data, since only relevant data should be counted when evaluating completeness, but not the quality of a datum or finding. In addition, metadata are required to make the meaning of the datum or finding clear, reducing uncertainty in a qualitative sense and facilitating reproducibility, and to assess the level of trust, reproducibility, repeatability, uncertainty and error. All of these issues affect the quality of a datum or finding. However, the quality of a datum or finding does not directly affect the completeness of the data. (5) The context determines the (meta)data required for completeness. Whilst quality is not dependent upon the intended use of the data, the specific (meta)data required for quality assessment may be dependent upon the experimental scenario. For example, specific kinds of (meta)data will be required in specific in vitro studies to assess assay interference and, hence, assess the error in a given datum. (6) The examples in this figure are by no means exhaustive or, necessarily, minimum requirements. The example contexts and their requirements are not necessarily mutually exclusive. For example, a nano-QSAR might be developed via integrating data across multiple in vitro mechanistic studies. (7) Where examples are provided in this figure of specific metadata which might be required for data completeness in different contexts, it should be recalled that the availability of these metadata could also affect the quality of individual data points or findings. Data regarding the surface composition and structure/ morphology are important.

Computational recommendations
In principle, when reporting data from any experimental study. N.B. The surface composition and structure/ morphology may arise due to a ligand shell/layer. Careful consideration of the extent to which completeness and quality assessment could be automated using these tools is required and may be contingent upon progress towards recommendation 6.3.2. Recommendation 6.3.3 is also pertinent here. 6.3.2 Standard templates for data exchange should be developed based upon the ISA-TAB-Nano specification.
Some early work towards this objective has already been carried out. The required templates are likely to be scenario specific. 6.3.3 Nanomaterial data resources providing completeness and quality scores should allow end-users to customise these based upon their own requirements.
The scoring systems should include the ability to customise and select the criteria upon which the degree of data completeness (in terms of fitness for purpose), or quality, is defined and provide the decision process and justification involved in this. The potential need to customise data completeness scoring primarily stems from the dependency of completeness on the use-case. The potential need to customise data quality scoring primarily stems from the lack of universal standards as to quality determination. resources, whilst other computational tools may need to be developed de novo.

Strategic recommendations
The proposals in Table 8 should be considered in order to develop scientific strategies for improving the manner in which the completeness and quality of nanomaterial data are evaluated in future work. Table 9 summarises recommendations regarding initiatives which could be undertaken by various organisations, in collaboration with the wider nanoscience community, to improve the manner in which the completeness and quality of nanomaterial data are evaluated.

Conclusions
The curation of nanomaterial data into electronic resources is crucial to realise the potential of nanotechnology to deliver benefits to society whilst having acceptable impacts upon human health and the environment. In order for these data to be fit for their intended purposes, they need to be sufficiently complete and of acceptable quality. Hence, appropriate evaluation of the quality and completeness of curated nanomaterial data is essential even if, in practice, analysis and conclusions may need to be drawn from imperfect data: such an evaluation can inform awareness of the limitations of any work based upon the available data. Any such evaluation needs to take account of the issues related to the completeness and quality of the underlying experimental data as well as additional issues related to their curation such as transcription errors. However, carrying out this evaluation in practice is non-trivial.
There are different perspectives as to exactly what these terms mean as well as different proposals as to how exactly the degree of completeness and quality of (curated) nanomaterial data should be evaluated in practice. After reviewing various existing proposals in light of broad and flexible definitions of these concepts, which accommodate the varying range of perspectives, more precise definitions are recommended to the nanoscience community. None of the existing proposals reviewed herein is perfect. A variety of challenges exist which impede appropriate evaluation of the completeness and quality of nanomaterial data. These challenges include the need to appropriately take account of the dependency of nano-material properties on their processing and storage history (i.e. time dependency), artefacts associated with biological testing of nanomaterials and incomplete understanding of which physicochemical properties and other experimental variables most significantly impact the effects of nanomaterials. In addition, the data requirements are likely to be dependent upon the precise experimental scenario (e.g. type of nanomaterials) and stakeholder requirements (e.g. regulatory decisions regarding a single nanomaterial vs. computational modelling). Some lessons might be learned from work in mature fields, such as the possibility of developing appropriate software tools to facilitate the efficient and transparent evaluation of (curated) experimental data. In the nanoscience domain, automated evaluation of data completeness and quality might best be supported via further development of nascent nanoinformatics resources. Common data collection templates based upon the ISA-TAB-Nano data exchange specification are envisaged. These will likely need to be adapted to the specific data requirements of different experimental scenarios and stakeholder objectives. The development of these resources will require community driven consensus regarding nanomaterial data requirements, which will best be supported by appropriate organisations and initiatives with an international reach. This article is one outcome of just such an initiative, the Nanomaterial Data Curation Initiative (NDCI), as reflected in the wide range of contributors and stakeholders who provided a variety of perspectives which informed the current work and resulted in a variety of recommendations to promote best practice and improve evaluation of the completeness and quality of (curated) nanomaterial data. An overview of the perspectives of these different stakeholders is presented in the ESI † of the current article.  Ongoing effort to support adoption and implementation will also be required, including by data curators.