Tag Archives: semantic technologies

What am I doing here? 3. Tabular Application Profiles⤴

The third thing (of four) that I want to include in my review of what projects I am working on is the Dublin Core standard for Tabular Application Profiles. It’s another of my favourites, a volunteer effort under the DCMI Application Profiles Working Group.

Application profiles are a powerful concept, central to how RDF can be used to create new data models without creating entirely new standards. The DCTAP draft says this about them:

An application profile defines metadata usage for a specific application. Profiles are often created as texts that are intended for a human audience. These texts generally employ tables to list the elements of the profile and related rules for metadata creation and validation. Such a document is particularly useful in helping a community reach agreement on its needs and desired solutions. To be usable for a specific function these decisions then need to be translated to computer code, which may not be a straightforward task.

About the work

We have defined an (extensible) set of twelve elements that can be used as headers in a table defining an application profile, a primer showing how they can be used, and along the way we wrote a framework for talking about metadata and application profiles. We have also worked on various implementations and are set to create a cookbook showing how DC TAP can be used in real world applications. The primer is the best starting point for understanding the output as a whole.

The framework for talking about metadata came about because we were struggling to be clear when we used terms like property or entity. Does “property” refer to something in the application profile or in the base standard or in as used in some metadata instance or does it refer to a property of some thing in the world? In short we decided that the things being described have characteristics and relationship to each other which are asserted in RDF metadata using statements that have a predicates in them, those predicates reference properties that are part of a pre-defined vocabulary, and an application profile defines templates for how the property is used in statements to create descriptions. There is a similar string of suggestions for talking about entities, classes and shapes as well as some comments on what we found too confusing and so avoid talking about. With a little care you can use terms that are both familiar in context and not ambiguous.

About my role

This really is a team effort, expertly lead by Karen Coyle, and I just try to help. I will put my hand up as the literal minded pedant who needed a framework to make sure we all understood each other. Otherwise I have been treating this a side project that gives me an excuse to do some python programming: I have documented my TAP2SHACL and related scripts on this blog, which focus on taking a DCMI TAP and expressing it as SHACL that can be used to validate data instances. I have been using this on some other projects that I am involved in, notably the work with PESC looking at how they might move to JSON-LD.

The post What am I doing here? 3. Tabular Application Profiles appeared first on Sharing and learning.

What am I doing here? 2. Open Competencies Network⤴

from Phil Barker @ Sharing and learning

I am continuing my January review of the projects that I am working on with this post about my work on the Open Competencies Network (OCN). OCN is a part of the T3 Network of Networks, which is an initiative of US Chamber of Commerce Foundation aiming to explore “emerging technologies and standards in the talent marketplace to create more equitable and effective learning and career pathways.” Not surprisingly the Open Competencies Network (OCN) focuses on Competencies, but we understand that term broadly, including any “assertions of academic, professional, occupational, vocational and life goals, outcomes … for example knowledge, skills and abilities, capabilities, habits of mind, or habits of practice” (see the OCN competency explainer for more). I see competencies understood in this way as the link between my interests in learning, education, credentials and the world of employment and other activities. This builds on previous projects around Talent Marketplace Signalling, which I also did for the US Chamber of Commerce Foundation.

About the work

The OCN has two working groups: Advancing Open Competencies (AOC), which deals with outreach, community building, policy and governance issues, and the Technical Advisory Workgroup. My focus is on the latter. We have a couple of major technical projects, the Competency Explorer and the Data Ecosystem Standards Mapping (DESM) Tool, both of which probably deserve their own post at some time, but in brief:

Competency Explorer aims to make competency frameworks readily available to humans and machines by developing a membership trust network of open registries each holding one or more competency frameworks and enabling search and retrieval of those frameworks and their competencies from any registry node in the network.

DESM was developed to support data standards organizations—and the platforms and products that use those standards—in mapping, aligning and harmonizing data standards to promote data interoperability for the talent marketplace (and beyond). The DESM allows for data to move from a system or product using one data standards to another system or product that uses a different data standard.

Both of these projects deal with heterogeneous metadata, working around the theme of interoperability between metadata standards.

About my role

My friend and former colleague Shiela once described our work as “going to meetings and typing things”, which pretty much sums up the OCN work. The purpose is to contribute to the development of the projects, both of which were initiated by Stuart Sutton, whose shoes I am trying to fill in OCN.

For the Competency Explorer I have helped turn community gathered use cases into features that can implemented to enhance the Explorer, and am currently one of the leads of an agile feature-driven development project with software developers at Learning Tapestry to implement as many of these features as possible and figure out what it would take to implement the others. I’m also working with data providers and Learning Tapestry to develop technical support around providing data for the Competency Explorer.

For DESM I helped develop the internal data schema used to represent the mapping between data standards, and am currently helping to support people who are using the tool to map a variety of standards in a pilot, or closed beta-testing. This has been a fascinating exercise in seeing a project through from a data model on paper, through working with programmers implementing it, to working with people as they try to use the tool developed from it.

The post What am I doing here? 2. Open Competencies Network appeared first on Sharing and learning.

DCAT AP DC TAP: a grown up example of TAP to SHACL⤴

from Phil Barker @ Sharing and learning

I’ve described a couple of short “toy” examples as proof of concept of turning a Dublin Core Application Profile (DC TAP) into SHACL in order to validate instance data: the SHACL Person Example and a Simple Book Example; now it is time to see how the approach fares against a real world example. I chose the EU joinup Data Catalog Application Profile (DCAT AP) because Karen Coyle had an interest in DCAT, it is well documented (pdf) with a github repo that has SHACL files, there is a Interoperability Test Bed validator for it (albeit a version late) and I found a few test instances with known errors (again a little dated). I also found the acronym soup of DCAT AP DC TAP irresistable.

You can see the extended TAP as Google sheets. The tap tab is the DCTAP, the other tabs are information about the Applicaiton profile, namespace prefixes and shapes that are needed by tap2shacl. You can also see the csv export files, test instances and generated SHACL for DCATAPDCTAP in my TAPExamples github repo. The actual TAP is maybe a little ragged around the edges. Partly I got bored, partly I wasn’t sure how far to go: for example, should every reference to a Catalog be a description that conforms to all the DCAT AP rules for a catalog, or is it sufficient to just have an IRI in the instance data? At the other extreme what to do about properties where the only requirement was that an entity of a certain type be referenced — should the SHACL demand the type be explicitly declared or is the intent that an IRI in the instance data is enough and the type may be inferred?

Reflections

I had to add a little functionality where I wanted a shape to be used against two targets, objects of a property and instances of a class. This actually prompted a fairly significant rewrite of how shape information is handled, and I have thought of further extensions.

It works in that valid SHACL is produced. The SHACL is more verbose than the hand crafted SHACL produced by the DCAT AP project, but I think that is to be expected from a general purpose conversion script. It also works in that when I run the test instances through the itb SHACL validator with my SHACL and through the DCAT AP specific validator they both flag the same errors. My shacl actually raises more error messages, but that is a result of it being more verbose, sometimes it gives two errors (one that the value of a predicate does not match the expecterd shape, the second about how the shape is not matched). The important thing is that the same fixes work on both.

I’m quite happy with this. There may be some requirements in the DCAT AP that I haven’t checked for, but this will do for now. Next I want to work out how best to require a specific skos concept scheme be used.

The only niggle that I have is that the TAP sheet in Google docs isn’t as easily human readable as the corresponding tables in the DCAT AP documentation. There’s a balance we’re working on between keep the necessary precision while weighing technical correctness against human friendliness, and it is hard to strike.

The post DCAT AP DC TAP: a grown up example of TAP to SHACL appeared first on Sharing and learning.

TAP to SHACL example⤴

from Phil Barker @ Sharing and learning

Last week I posted Application Profile to Validation with TAP to SHACL about converting a DCMI Tabular Application Profile (DC TAP) to SHACL in order to validate instance data. I ended by saying that I needed more examples in order to test that it worked: that is, not only check that the SHACL is valid, but also that validates / raises errors as expected when used with instance data.

SHACL Person Example

Here’s a simple example of a person and their employer, drawn from the example used in the SHACL spec. The TAP for this only has three lines:

Line 1: Person data may have zero or one ex:ssn property for Social Security Number, which must be a string matching the regular expression for nnn-nn-nnnn (n = any digit)

Line 2: Person data may have zero or more ex:worksFor properties with an IRI referencing a node that matches the CompanyShape.

Line 3: Company data must have one and only one occurrence of the rdf:type property with the value ex:Company (or equivalent IRI).

And the extended TAP Shapes csv has two:

Line 1: nodes of class ex:Person must match the PersonShape, which is closed, though any occurrence of rdf:type is ignored, any failure to match is a Violation of the application profile.

Line 2: object nodes of the ex:worksFor property must match the CompanyShape, which is open, any failure to match is a Violation of the application profile.

The SHACL produced is

# SHACL generated by python AP to shacl converter
@base <http://example.org/> .
@prefix ex: <http://example.org/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<PersonShape> a sh:NodeShape ;
    sh:closed true ;
    sh:description "Node shape for person class"@en ;
    sh:ignoredProperties ( rdf:type ) ;
    sh:name "Person Shape"@en ;
    sh:property <personshapeEmployer>,
        <personshapeSocialSecurityNumber> ;
    sh:targetClass <Person> .

<CompanyShape> a sh:NodeShape ;
    sh:class <Company> ;
    sh:closed false ;
    sh:description "Shape for employers"@en ;
    sh:name "Employer"@en ;
    sh:targetObjectsOf <woksFor> .

<personshapeEmployer> a sh:PropertyShape ;
    sh:name "Employer"@en-US ;
    sh:node <CompanyShape> ;
    sh:nodeKind sh:IRI ;
    sh:path <worksFor> .

<personshapeSocialSecurityNumber> a sh:PropertyShape ;
    sh:datatype xsd:string ;
    sh:maxCount 1 ;
    sh:name "Social Security Number"@en-US ;
    sh:nodeKind sh:Literal ;
    sh:path <ssn> ;
    sh:pattern "^\\d{3}-\\d{2}-\\d{4}$" .

Results

This isn’t the same SHACL as in the spec, but when used with instance data (sample data in github, three are invalid taken from the spec, and one is valid data) in the Join-up itb SHACL validator it gives the same result as expected:

I had to make one change to the program to get this working, adding an “ignore props” column to the shape information sheet, which is then processed into sh:ignoredProperties.

I guess it is worth noting that the SHACL generated from tap2shacl is always more verbose than hand written examples: some defaults are always explicit, and property shapes are always identified objects in their own block rather than bnodes in the NodeShape. That can make the output less human readable. Also, as I currently re-use property shapes that recur on different node shapes, there can be unnecessary repetition. There is a mechanism in TAP for setting “defaults” which might be useful when, say, all instances of dct:description must follow the same rules whatever class they are used on.

My next example will (hopefully) be based on a well known real-world application profile; a lot longer, and introducing some more complex rules. Watch this space.

The post TAP to SHACL example appeared first on Sharing and learning.

Application Profile to Validation with TAP to SHACL⤴

from Phil Barker @ Sharing and learning

Over the past couple of years or so I have been part of the Dublin Core Application Profile Interest Group creating the DC Tabular Application Profile (DC-TAP) specification. I described DC-TAP in a post about a year ago as a “human-friendly approach that also lends itself to machine processing”, in this post I’ll explore a little about how it lends itself to machine processing.

A metadata application profile represents an implementor’s or a community’s view on how a metadata description should be formed: which terms should be used from which vocabularies, are they required or optional? may they be repeated? what range of values do expect each of them to have? and so on [see Heery and Patel and the Singapore Framework for more details]. Having created an application profile it is fairly natural to want to ask whether specific instance data conforms with it — does it have the data required in the form required by the implementor or community’s application(s)? I have been using the EU-funded joinup Interoperability Test Bed‘s SHACL Validator for this, and so I would like to generate SHACL for my validation.

The simple book application profile.

There are several current and potential application profiles that I am interested in, including the Credential Engine minimum data requirements for using CTLD to describe the various types of resource in the Credential Registry. I also have a long standing interest in application profiles of schema.org and LRMI for describing learning resources, which is currently finding an outlet in the IEEE P2881 Standard for Learning Metadata working group. But these and any other real application profile tend to be long and can be complex. A really short simple application profile is more useful for illustrative purposes, and we in the DC-TAP group have been using various iterations around a profile describing books (even for such a simple application profile the table doesn’t display well with my blog theme, so please take a look at it on github). Here’s a summary of what it is intended to encode:

Line 1: Book instance data must have one and only one dct:title of type rdf:langString.
Line 2: Book instance data may have zero or more dct:creator described as an RDF instance with a URI or a BNODE, matching the #Author shape.
Line 3: Book instance data may have zero or one sdo:isbn with Literal value being an xsd:string composed of 13 digits only.
Line 4: Book instance data must have rdf:type of sdo:Book.
Line 5: Author instance data may have zero or more foaf:givenName with Literal value type xsd:string.
Line 6: Author instance data may have zero or more foaf:familyName with Literal value type xsd:string
Line 7: Author instance data must have rdf:type of foaf:Person

(Let’s leave aside any questions of whether those are sensible choices, OK?)

A SHACL view of the TAP

Looking at the TAP through the lens of SHACL, we can draw a parallel between what in TAP we call statement constraints, i.e. the rows in the TAP, which each identify a property and any rules that constrain its use, and what SHACL calls property shapes. Likewise what in TAP we call shapes (a group of statement constraints) aligns with what SHACL calls a Node Shape. The elements of a statement constraint map more or less directly to various SHACL properties that can be used with Node and Property Shapes. So:

Construct in TAP	Rough equivalent in SHACL
Statement Constraint	Property Shape
Shape	Node Shape
propertyID	sh:path of a sh:PropertyShape
propertyLabel	sh:name on a sh:PropertyShape
mandatory = TRUE	sh:minCount = 1
repeatable = FALSE	sh:maxCount = 1
valueNodeType	sh:nodeKind
valueDataType	sh:dataType
valueShape	sh:node
valueConstraint	depends on valueConstraintType and valueNodeType

Processing valueConstraints can get more complex than other elements. If there is no valueNodeType and a single entry, then it is used as the value for sh:hasValue, either as a Literal or IRI depending on valueNodeType. If the valueConstraintType is “pickList” or if there are more than one entries in valueConstraint then you need to create a list to use with the sh:or property. If the valueConstraintType is “pattern” then the mapping to sh:pattern is straightforward, and that should only apply to Literal values. Some other SHACL constraint components are not in the DCTAP core set of suggested entries for valueConstraint, but it seems obvious to add “minLength”, “maxLength” and so on to correspond to sh:minLength, sh:maxLength etc.; lengthRange works too if you specify the range in a single entry (I use the format “n..m”). I do not expect DC TAP will cover all of what SHACL covers, so don’t expect to find sh:qualifiedValueShape or other complex constraint components.

As alluded to above, TAP allows for multiple entries in a table cell to provide alternative ways of fulfilling a constraint. These lists of entries need to be processed in different ways depending on which element of statement constraint they relate to. Often they need turning into a list to be used with sh:or or sh:in; lists of alternative valueNodeType need turning into the corresponding value for sh:nodeType, for example IRI BNODE becomes sh:BlankNodeOrIRI.

Extended TAPs to provide other SHACL terms

Some things needed by SHACL (and other uses of application profiles) are not in DC TAP. We know that TAP only covers a core and we expect different solutions to providing other information to emerge depending on context. For some people providing some of the additional information as, say, a YAML file will work; for other people or other data, further tables may be preferable. So while we know that metadata about the application profile and a listing of the IRI stems for any prefixes used for compact URI encodings in the TAP need to be provided, we don’t specify how. I chose to use additional tables for all this data.

I’ve already touched on how additional constraints useful in SHACL, like minLength that are easily provided if we extend the range of allowed valueConstraintType. Another useful SHACL property is sh:severity, for this I added an extra column to the TAP table.

However the biggest omission from the TAP of data useful in SHACL is data about sh:NodeShapes. From the “Shape” column we know which shapes the properties relate to, but we have no way of providing descriptions for these shapes, or, most crucially, specifying what entities in the instance data should conform to which shapes. I use a table to provide this data as well. Currrently(*), it has columns for shapeID, label, comment, target, targetType (these last two can be used to set values of sh:targetObjectsOf, sh:targetClass, sh:targetNode, sh:targetSubjectsOf as appropriate), closed (true or false, to set sh:Closed), mandatory, severity and note. (*This list of columns is somewhat in flux, and the program described below doesn’t process all of them).

A Google Sheets template for extended TAPs

So, for my application profile I have four tables: the DC TAP, plus tables for: metadata about the application profile; prefixes/namespaces used; information about TAP Shapes / sh.NodeShapes. All these tables can be generated as spreadsheet tabs in a single workbook. The details are still a little fluid, but I have created a template for such in Google Sheets, which also includes some helpful additions like data validation to make sure that the values in cells make sense. You are welcome to make a copy of this template and try it out, it includes a sheet with further information and details of how to leave feedback.

Processing TAP to SHACL

I have also written a set of python classes to convert CSV files exported from the extended TAP into SHACL. These are packaged together as tap2shacl, available on Github — it is very much an alpha-release proof of concept, that doesn’t cover all the things discussed above let alone some of the things I have glossed over, but feel free to give it a try.

The architecture of the program is as follows:

I have data classes for the application profile as a whole and individual property statements. This includes methods to read in Metadata, shape information and prefix/namespace information from CSV files as exported from Google Sheets (or other editor with the same column headings)
I use Tom Baker’s dctap-python program to read the CSV of the TAP. This does lots of useful validity checking and normalization as well as handling a fair few config options, and generally handles the variation in CSVs better than the methods I wrote for other tables. TAP2AP handles the conversion from Tom’s format to my python AP classes.
The AP2SHACL module contains a class and methods to convert the python AP classes into a SHACL RDF Graph and serialize these for output (leaning heavily on rdf-lib).
Finally the TAP2SHACL package pulls these together and provides a command line interface.

If that seems like more modules and github repos that necessary, you may be right, but I wanted to be hyper-granular because I have use cases where the input isn’t a TAP CSV and the output isn’t SHACL. For example, the Credential Engine minimum data requirements are encoded in JSON, for other sources YAML is another possibility, and I have ideas about converting Application Profile diagrams; and I can see sense in outputting ShEx and JSON-Schema as well as SHACL. I also want to keep the number of imported third-party modules down to a minimum: why should someone wanting to create JSON-Schema have to import the rdf-lib classes needed for SHACL?

Does it work?

Well, it wouldn’t have been fair to let you read this far if the answer wasn’t broadly “yes” Checking that it works is another matter.

There are plenty of unit tests in the code for me to be confident that it can read some files and output some RDF, albeit with the caveat that it is alpha-release software so it’s not difficult to create a file that it cannot read, often because the format is not quite right.

There are even some integration tests so that I know that the RDF output from some TAPs matches the valid SHACL that I expect, at least for simple test cases. Again, it is not difficult to generate invalid SHACL or not produce the terms you would expect if there happens to be something in the TAP that I haven’t yet implemented. TAP is quite open, and the software is still developing, so I’ll not attempt to list the potential mismatches here, but I’ll be working documenting them in github issues.

But then there’s the question of whether the SHACL that I expect correctly encodes the rules in the application profile. That takes testing itself, so for each application profile I work on I need test cases of instance data that either matches or doesn’t match the expectations in mind when creating the application profile. I have a suite of 16 test cases for the simple book profile. These can be used in the SHACL Validator with the SHACL file generated from the TAP; and yes, mostly they work.

I have got to admit that I find the implications of needing 16 tests for such a simple profile somewhat daunting when thinking about real-world profiles that are an order of magnitude or more larger, but I hope that confidence and experience built with simple profiles will reduce the need to so many test cases. So my next steps will be to slowly build up the range of constraints and complexity of examples. Watch this space for more details, or contact me if you have suggestions.

The post Application Profile to Validation with TAP to SHACL appeared first on Sharing and learning.

SHACL, when two wrongs make a right⤴

from Phil Barker @ Sharing and learning

I have been working with SHACL for a few months in connexion with validating RDF instance data against the requirements of application profiles. There’s a great validation tool created as part of the JoinUp Interoperability Test Bed that lets you upload your SHACL rules and a data instance and tests the latter against the former. But be aware: some errors can lead to the instance data successfully passing the tests; this isn’t an error with the tool, just a case of blind logic: the program doing what you tell it to regardless of whether that’s what you want it to do.

The rules

Here’s a really minimal set of SHACL rules for describing a book:

@base <http://example.org/shapes#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns> .
@prefix sdo: <https://schema.org/> .
@prefix sh: <http://www.w3.org/ns/shacl#> .

<Book_Shape> a sh:NodeShape ;
    sh:property
        <bookTitle_Shape> ;
    sh:targetClass sdo:Book .

<bookTitle_Shape> a sh:PropertyShape ;
    sh:path dct:title ;
    sh:datatype rdf:langString ;
    sh:nodeKind sh:Literal ;
    sh:minCount 1 ;
    sh:severity sh:Violation .

Essentially it says that the description of anything typed as a schema.org:Book should have a title provided as a langString using the Dublin Core title property. Here’s some instance data

@prefix : <http://example.org/books#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix sdo: <https://schema.org/> .

:42 a sdo:Book;
  dct:title "Life the Universe and Everything."@en .

Throw those into the SHACL validator and you get:

Result: SUCCESS
Errors: 0
Warnings: 0
Messages: 0

Which (I think) is what you should get.

So wrong it’s right

But what about this instance data:

@prefix : <http://example.org/books#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix sdo: <http://schema.org/> .

:42 a sdo:Book;
  dct:title "Life the Universe and Everything." .

For this the validator also returns

Result: SUCCESS
Errors: 0
Warnings: 0
Messages: 0

Did you spot the difference? Well, for one thing the title has no language attibute, it’s not a langString.

How about:

@prefix : <http://example.org/books#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix sdo: <http://schema.org/> .

:42 a sdo:Book .

which has no title at all, but still you’ll get the “success, no errors” result. That cannot be right, can it?

Well, yes, it is right. You see there is another error in those last two examples. In the SHACL and the first example, the URI for Schema.org is given as

@prefix sdo: <https://schema.org/> .

In the second and third examples it is

@prefix sdo: <http://schema.org/> .

That’s the sort of subtle and common mistake I would like to pick up, but in fact it stops the validator from picking up any mistakes. That’s because the SHACL rules apply to anything typed as a https://schema.org/Book and in the second and third examples (where the prefix is http, not https) there isn’t anything typed as such. No rules apply, no rules are broken: zero errors — success!

What to do?

I’m not really sure. I see this type of problem quite a lot (especially if you generalize “this type of problem” to mean the test doesn’t have quite the logic I thought it did). I suppose lesson one was always to test shapes with invalid data to make sure they work as expected. That’s a lot of tests.

Arising from that is to write SHACL rules to check for everything: the errors above would be picked up if I had checked to make sure there is always one entity with the expected type for Books: there’s recipe for this on the SHACL wiki.

Generalizing on the idea that simple typos can mean data not being tested because the term identifier doesn’t match any term in the schema your using, it’s worth checking that all term identifiers in the instance data and the SHACL are actually in the schema. This will pickup when sdo:Organisation is used instead of sdo:Organization. SHACL won’t do this, but it’s easy enough to write a python script that does.

The post SHACL, when two wrongs make a right appeared first on Sharing and learning.

When RDF breaks records⤴

from Phil Barker @ Sharing and learning

In talking to people about modelling metadata I’ve picked up on a distinction mentioned by Staurt Sutton between entity-based modelling, typified by RDF and graphs, and record-based structures typified by XML; however, I don’t think making this distinction alone is sufficient to explain the difference, let alone why it matters. I don’t want to get into the pros and cons of either approach here, just give a couple of examples of where something that works in a monolithic, hierarchical record falls apart when the properties and relationships for each entity are described separately and those descriptions put into a graph. These are especially relevant when people familiar with XML or JSON start using JSON-LD. One of the great things about JSON-LD is that you can use instance data as if it were JSON, without really paying much regard to the “LD” part; that’s not true when designing specs because design choices that would be fine in a JSON record will not work in a linked data graph.

1. Qualified Instances

It’s very common in a record-oriented approach when making statements about something that many people may have done, such as attending a specific school, earning a qualification/credential, learning a skill etc, to have a JSON record that looks something like:

{ "studentID": "Person1",
  "studentName": "Betty Rizzo",
  "schoolsAttended": [
    { "schoolID": "School1",
      "schoolName": "Rydell High School",
      "schoolAddress" : {...}
      "startDate": "1954",
      "endDate": "1959"
    }
  ]
}

It’s tempting to put a @context on the top of this to map the property keys to an RDF vocabulary and call it linked data. That’s sub-optimal. To see why consider two students Betty, as above, and Sandy who joined the school for her final academic year, 1958-59. Representing her data and Betty’s as RDF graphs we would get something like:

The upper graph is a representation of what you might get for the record about Rizzo shown above, if you choose a suitable @context. The lower is similar data about Sandy. When this data is loaded into an RDF triple store, the statements will be stored separately, and duplicates removed. We can show that data as a single merged graph:

Whereas in a record the hierarchy preserves the scope for statements like startDate and endDate so that we know who they refer to, in the RDF graph statements from the JSON object describing the school attended are taken as being about the school itself. The problem arises because the information about the school is treated as data that can be linked to by anything that relates to the school, not just the entity in whose record it was found, which makes sense in terms of data management.

There are options for fixing this: one is not to merge the graphs about the Betty and Sandy, but that means repeating all the data about the school in every record that mentions it; another possible solution is to use the property-graph or RDF-star approach of annotating the schoolAttended property directly with startDate and endDate; but often the answer lies in better RDF modelling. In this case we could create an entity to model the attendance of a person at a school:

and when these are merged:

which keeps the advantage of not duplicating information about the school while maintaining the information about who attended which school when. In JSON-LD this conbined graph would look something like

{ "@context": {...},
  "@graph": [
    { "@id": "Person1",
      "name": "Betty Rizzo",
      "schoolAttended": { 
        "startDate": "1953",
        "endDate": "1959",
        "at": {"@id": "School1"}
      }
    },{
      "@id": "Person2",
      "name": "Sandy Olsson",
      "schoolAttended": {
        "startDate": "1953",
        "endDate": "1959",
        "at": {"@id": "School1"}
      }
   },{
     "@id": "School1",
     "name": "Rydell High",
     "address": {
       "@type": "PostalAddress",
       "...": "..."
     }
   }]
}

Finally, those who just want a JSON record for an individual student that could easily be converted to LD could use something like:

 
{ "studentID": "Person1", 
  "schoolsAttended": [ 
    { "startDate": "1954", 
      "endDate": "1959",
      "at": {
          "schoolID": "School1",
          "schoolName": "Rydell High School",
          "schoolAddress" : {...}
      } 
  ] 
}

You might think that the “attendance” object sitting between a person and the school is a bit artificial and unintuitive, which it is, but it’s no worse than the tables that RDBM systems need for many-to-many relationships.

2. Lists

Another pattern that comes up a lot is when logically separate resource may be ordered in different ways for different reasons. This may be people in a queue, journal articles in a volume, or learning resources in a larger learning opportunity; anywhere that you might want to say “this” comes before “that”. Say we have an educational program that has a number of courses in it that should be taken in sequential order. JSON lists are ordered, so as a record this seems to work:

{
  "name": "My Program",
  "hasCourse": [
    {"name": "This"},
    {"name": "That"},
    {"name": "The other"}
  ]
}

So we sprinkle on some syntactic sugar for JSON-LD:

{
 "@context": {"@vocab": "http://schema.org/", 
               "@base": "http://example.org/resources/"},
 "@type": "EducationalOccupationalProgram",
 "name": "My Program",
 "hasCourse": [
   {"@type": "Course",
    "@id": "Course1",
    "name": "This"},
   {"@type": "Course",
    "@id": "Course2",
    "name": "That"},
   {"@type": "Course",
    "@id": "Course3",
    "name": "The other"}
 ]
}

But there is no RDF statement in there about ordering, and the ordering of JSON’s arrays is not preserved in other RDF syntaxes (unless there is something in the @context to say the value of hasCourse is an @list, it wouldn’t be appropriate to say that every value of hasPart is an ordered list because not every list of parts will be an ordered list). So if we convert the JSON-LD into triples and store them, there is no saying how to order the results returned by a query.

The simple solution would be to have a property to state the position of the course in an ordered list (schema.org/position is exactly this)—but don’t be too hasty: if these courses are taken in more than one program, is Course 2 always going to be second in the sequence? Probably not. In general when resources are reused in different contexts they will probably be used in different orders, “this” may not always come before “that”. That’s why the ordering is best specified at one remove from resources themselves. For example, one of the suggestions for ordering content in K12-OCX is to create a table of contents as an ordered list of items that point to the content, something like:

{
  "@context": {
    "@vocab": "http://schema.org/",
    "ocx": "http://example.org/ocx/",
    "@base": "http://example.org/resources/",
    "item": {"@type": "@id"}
  },
  "@type": "EducationalOccupationalProgram",
  "name": "My Program",
  "ocx:hasToC": {
    "@type": "ItemList",
    "name": "Table of Contents",
    "itemListOrder": "ItemListOrderAscending",
    "numberOfItems": "3",
    "itemListElement": [
      { "@type": "ListItem",
        "item": "Course1",
        "position": 1},
      { "@type": "ListItem",
        "item": "Course2",
        "position": 2 },
      { "@type": "ListItem",
        "item": "Course3",
        "position": 3 }
    ]
  },
 "hasCourse": [
   {"@type": "Course",
    "@id": "Course1",
    "name": "This"},
   {"@type": "Course",
    "@id": "Course2",
    "name": "That"},
   {"@type": "Course",
    "@id": "Course3",
    "name": "The other"}
 ]
}

or if you prefer to use built-in RDF constructs there is that @list option:

{ "@context": {
    "@vocab": "http://schema.org/", 
    "ocx": "http://example.org/ocx/", 
    "@base": "http://example.org/resources/", 
    "ocx:hasToC": {"@container": "@list"}
  },
  "@type": "EducationalOccupationalProgram",
  "@id": "Program",
  "name": "My Program",
  "ocx:hasToC": ["Course1", "Course2", "Course3"],
  "hasCourse": [
  { "@id": "Course1",
    "@type": "Course",
    "name": "this"
  },{
    "@id": "Course2",
    "@type": "Course",
    "name": "that"
  },{
    "@id": "Course3",
    "@type": "Course",
    "name": "the other"
  }]
}

When this is processed by something like JSON-LD playground you will see that the list of values for hasToC is replaced by a set of statements about blank-nodes which mean this comes before the others:

<ocx:hasToC> _:b0 .
_:b0 <rdf:first> "http://example.org/resources/Course1" .
_:b0 <rdf:rest> _:b1 .
_:b1 <rdf:first> "http://example.org/resources/Course2" .
_:b1 <rdf:rest> _:b2 .
_:b2 <rdf:first> "http://example.org/resources/Course3" .
_:b2 <rdf:rest> <rdf:nil> .

Conclusion

If you’ve made it this far you deserve the short summary advice. The title for this post was meant literally. Representing a record in RDF will break the record down into separate statements, each about one thing, each saying one thing, with the assumption that those statements are each valid on their own. In modelling for JSON-LD you need to make sure that everything you say about an object is true even when that object is separated from the rest of the record.

The post When RDF breaks records appeared first on Sharing and learning.

Strings to things in context⤴

from Phil Barker @ Sharing and learning

As part of work to convert plain JSON records to proper RDF in JSON-LD I often want to convert a string value to a URI that identifies a thing (real world concrete thing or a concept).

Simple string to URI mapping

Given a fragment of a schedule in JSON

{"day": "Tuesday"}

As well as converting "day" to a property in an RDF vocabulary I might want to use a concept term for “Tuesday” drawn from that vocabulary. JSON-LD’s @context lets you do this: the @vocab keyword says what RDF vocabulary you are using for properties; the @base keyword says what base URL you are using for values that are URIs; the @id keyword maps a JSON key to an RDF property; and, the @type keyword (when used in the @context object) says what type of value a property should be, the value of @type that says you’re using a URI is "@id" (confused by @id doing double duty? it gets worse). So:

{
  "@context": {
    "@vocab": "http://schema.org/",
    "@base": "http://schema.org/",
    "day": {
       "@id": "dayOfWeek",
       "@type": "@id"
    }
  },
  "day": "Tuesday"
}

Pop this in to the JSON-LD playground to convert it into N-QUADS and you get:

_:b0 <http://schema.org/dayOfWeek> <http://schema.org/Tuesday> .

Cool.

What type of thing is this?

The other place where you want to use URI identifiers is to say what type/class of thing you are talking about. Expanding our example a bit, we might have

{
  "type": "Schedule",
  "day": "Tuesday"
}

Trying the same approach as above, in the @context block we can use the @id keyword to map the string value "type" to the special value "@type"; and, use the @type keyword with special value "@id" to say that the type of value expected is a URI, as we did to turn the string “Tuesday” into a schema.org URI. (I did warn you it got more confusing). So:

{
  "@context": {
    "@vocab": "http://schema.org/",
    "@base": "http://schema.org/",
    "type": {
       "@id": "@type",
       "@type": "@id"    
    },
    "day": {
       "@id": "dayOfWeek",
       "@type": "@id"
    }
  },
  "type": "Schedule",
  "day": "Tuesday"
}

Pop this into the JSON-LD playground and convert to N-QUADS and you get

_:b0 <http://schema.org/dayOfWeek> <http://schema.org/Tuesday> .
_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://schema.org/Schedule> .

As we want.

Mixing it up a bit

So far we’ve had just the one RDF vocabulary, say we want to use terms from a variety of vocabularies. For the sake of argument, let’s say that no one vocabulary is more important than another, so we don’t want to use @vocab and @base to set global defaults. Adding another term from a custom vocab in to the our example:

{ 
  "type": "Schedule",
  "day": "Tuesday",
  "onDuty": "Phil" 
}

In the context we can set prefixes to use instead of full length URIs, but the most powerful feature is that we can use different @context blocks for each term definition to set different @base URI fragments. That looks like:

{
  "@context": {
    "schema": "http://schema.org/",
    "ex" : "http://my.example.org/",
    "type": {
       "@id": "@type",
       "@type": "@id",
       "@context": {
         "@base": "http://schema.org/"        
      }
    },
    "day": {
      "@id": "schema:dayOfWeek",
      "@type": "@id",
      "@context": {
         "@base": "http://schema.org/"        
      }
    },
   "onDuty": {
     "@id": "ex:onDuty",
       "@type": "@id",
       "@context": {
         "@base": "https://people.pjjk.org/"
      }
    }
  },
  "type": "Schedule",
  "day": "Tuesday",
  "onDuty": "phil"
}

Translated by JSON-LD Playground that gives:

_:b0 <http://my.example.org/onDuty> <https://people.pjjk.org/phil> .
_:b0 <http://schema.org/dayOfWeek> <http://schema.org/Tuesday> .
_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://json-ld.org/playground/Schedule> .

Hmmm. The first two lines look good. The JSON keys have been translated to URIs for properties from two different RDF vocabularies, and their string values have been translated to URIs for things with different bases, so far so good. But, that last line: the @base for the type isn’t being used, and instead JSON-LD playground is using its own default. That won’t do.

The fix for this seems to be not to give the @id keyword for type the special value of "@type", but rather treat it as any other term from an RDF vocabulary:

{
  "@context": {
    "schema": "http://schema.org/",
    "ex" : "http://my.example.org/",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "type": {
       "@id": "rdf:type",
       "@type": "@id",
       "@context": {
         "@base": "http://schema.org/"        
      }
    },
    "day": {
      "@id": "schema:dayOfWeek",
      "@type": "@id",
      "@context": {
         "@base": "http://schema.org/"        
      }
    },
   "onDuty": {
     "@id": "ex:onDuty",
       "@type": "@id",
       "@context": {
         "@base": "https://people.pjjk.org/"
      }
    }
  },
  "type": "Schedule",
  "day": "Tuesday",
  "onDuty": "phil"
}

Which gives:

_:b0 <http://my.example.org/onDuty> <https://people.pjjk.org/phil> .
_:b0 <http://schema.org/dayOfWeek> <http://schema.org/Tuesday> .
_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://schema.org/Schedule> .

That’s better, though I do worry that the lack of a JSON-LD @type key might bother some.

Extensions and Limitations

The nested context for a JSON key works even if the value is an object, it can be used to specify the @vocab and @base and any namespace prefixes used in the keys and values of the value object. That’s useful if title in one object is dc:title and title in another needs to be schema:title.

Converting string values to URIs for things like this is fine if the string happens to match the end of the URI that you want. So, while I can change the a JSON key "author" into the property URI <https://www.wikidata.org/prop/direct/P50> I cannot change the value string "Douglas Adams" into <https://www.wikidata.org/entity/Q42>. For that I think you need to use something a bit more flexible, like RML, but please comment if you know of a solution to that!

Also, let me know if you think the lack of a JSON-LD @type keyword, or anything else shown above seems problematic.

The post Strings to things in context appeared first on Sharing and learning.

JDX: a schema for Job Data Exchange⤴

from Phil Barker @ Sharing and learning

[This rather long blog post describes a project that I have been involved with through consultancy with the U.S. Chamber of Commerce Foundation. Writing this post was funded through that consultancy.]

The U.S. Chamber of Commerce Foundation has recently proposed a modernized schema for job postings based on the work of HR Open and Schema.org, the Job Data Exchange (JDX) JobSchema+. It is hoped JDX JobSchema+ will not just facilitate the exchange of data relevant to jobs, but will do so in a way that helps bridge the various other standards used by relevant systems. The aim of JDX is to improve the usefulness of job data including signalling around jobs, addressing such questions as: what jobs are available in which geographic areas? What are the requirements for working in these jobs? What are the rewards? What are the career paths? This information needs to be communicated not just between employers and their recruitment partners and to potential job applicants, but also to education and training providers, so that they can create learning opportunities that provide their students with skills that are valuable in their future careers. Job seekers empowered with greater quantity and quality of job data through job postings may secure better-fitting employment faster and for longer duration due to improved matching. Preventing wasted time and hardship may be particularly impactful for populations whose job searches are less well-resourced and those for whom limited flexibility increases their dependence on job details which are often missing, such as schedule, exact location, and security clearance requirement. These are among the properties that JDX provides employers the opportunity to include for easy and quick identification by all. In short, the data should be available to anyone involved in the talent pipeline. This broad scope poses a problem that JDX also seeks to address: different systems within the talent pipeline data ecosystem use different data standards so how can we ensure that the signalling is intelligible across the whole ecosystem?

The starting point for JDX was two of the most widely used data standards relevant to describing jobs: HR Open Standards Recruiting standard, part of the foremost suite of standards covering all aspects of the HR sector and the schema.org JobPosting schema, which is used to make data on web pages accessible to search engines, notably Google’s Job Search. These, and an analysis of the information required around jobs, job descriptions and job postings, their relationships to other entities such as organizations, competencies, credentials, experience and so on, were modelled in RDF to create a vocabulary of classes, properties, and concept schemes that can be used to create data. The full data model, which can be accessed on GitHub, is quite extensive: the description of jobs that JDX enables goes well beyond what is required for a job posting advertising a vacancy. A subset of the full model comprising those terms useful for job postings was selected for pilot testing, and this is available in a more accessible form on the Chamber Foundation’s website and is documented on the Job Data Exchange website. The results of the data analysis, modelling and piloting were then fed back into the HR Open and schema.org standards that were used as a starting point.

This is where things start to get a little complicated, as it means JDX has contributed to three related efforts.

JobPostings in schema.org

The modelling and piloting highlighted and addressed some issues that were within schema.org’s scope of enabling the provision of structured data about job postings on the web. These were discussed through a W3C Community Group on Talent Marketplace Signalling, and the solutions were reconciled with schema.org’s wider model and scope as a web-wide vocabulary that covers many other types of things apart from Jobs. The outcomes include that schema.org/JobPosting has several new properties (or modifications to how existing properties are used) allowing for such things as: a job posting with more than one vacancy, a job posting with a specified start date, a job posting with requirements other than competencies — i.e. physical, sensory and security clearance requirements, and more specific information about contact details and location within the company structure for the job being advertised.

Because schema.org and JDX are both modelled in RDF as sets of terms that can be used to make independent statements about entities (rather than a record-based model such as XML documents) it was relatively easy to add terms to schema.org that were based on those in JDX. The only reason that the terms added to schema.org are not exactly the same as the terms in JDX JobSchema+ is that it was sometimes necessary to take into account already existing properties in schema.org, and the wider purpose and different audience of schema.org.

JDX in HROpen

As with schema.org, JDX highlighted some issues that are within the scope of the HROpen Standards Recruiting standard, and the aim is to incorporate the lessons learnt from JDX into that standard. However the Recruiting standard is part of the inter-linked suite of specifications that HROpen maintains across all aspects of the HR domain, and these standards are in plain JSON, a record-based format specified through JSON-Schema files not RDF Schema. This makes integration of new terms and modelling approaches from JDX into HROpen more complicated than was the case with schema.org. As a first step the property definitions have been translated into JSON-Schema, and partially integrated into the suite of HROpen standards, however some of the structures, for example for describing Organizations, were significantly different to how other HROpen standards treat the same types of entity, and so these were kept separate. The plan for the next phase is to further integrate JDX into the existing standards, enhance the use cases and documentation and include RDF, JSON Schema, and XML XSD.

JDX JobPosting+ RDF Schema

Finally, of course, JDX still exists as an RDF Schema, currently on github. The work on integration with HROpen surfaced some errors and other issues, which have been addressed. Likewise feeding back into schema.org JobPosting means that there are new relationships between terms in JDX and schema.org that can be encoded in the JDX schema. Finally there is potential for other changes and remodelling as a result of findings from the JDX pilot of job postings. But given the progress made with integrating lessons learnt into schema.org and the HROpen Recruiting standard, what is the role of the RDF Schema compared to these other two?

Standard Strengths and Interoperability

Each of the three standards has strengths in its own niche. Schema.org provides a widely scoped vocabulary, mostly used for disseminating information on the open web. The most obvious consumers of data that use terms from schema.org are search engines trying to make sense of text in web pages, so that they can signal the key aspects of job postings with less ambiguity than can easily be done by processing natural text. Of course such data is also useful for any system that tries to extract data from webpages. Schema.org is also widely used as a source of RDF terms for other vocabularies, after all it doesn’t make much sense for every standard to define its own version of a property for the name of the thing being described, or a textual description of it—more on this below in the discussion of harmonization.

HROpen Standards are designed for system-to-system interoperability within the HR domain. If organization A and organization B (not to mention organizations C through to Z) have systems that do the same sort of thing with the same sort of data, then using an agreed standard for the data they care about clearly brings efficiencies by allowing for systems to be designed to a common specification and for organizations to share data where appropriate. This is the well understood driving force for interoperability specifications.

it is useful to have a common set of “terms” from which data providers can pick and choose what is appropriate for communicating different aspects of what they care about

But what about when two organizations are using the same sort of data for different things? For example, it might be that they are part of different verticals which interact with each other but have significant differences aside from where they overlap; or it might be that one organization provides a horizontal service, such as web search, across several verticals. This is where it is useful to have a common set of “terms” from which data providers can pick and choose what is appropriate for communicating different aspects of what they care about to those who provide services that intersect or overlap with their own concern. For example a fully worked specification for learning outcomes in education would include much that is not relevant to the HR domain and much that overlaps; furthermore HR and education providers use different systems for other aspects of their work: HR will care about integration with payroll systems, education about integration with course management systems. There is no realistic prospect that the same data standards can be used to the extent that the record formats will be the same; however with the RDF approach of entity-focused description rather than defining a single record structure, there is no reason why some of the terms that are used to describe the HR view of competency shouldn’t also be used to describe the education view of learning outcomes. Schema.org provides a broad horizontal layer of RDF terms that can be used across many domains; JDX provides a deeper dive into the more specific vocabulary used in jobs data.

Data Harmonization

This approach to allowing mutual intelligibility between data standards in different domains to the extent that the data they care about overlaps (or, for that matter, competing data standards in the same domain) is known as data harmonization. RDF is very much suited to harmonization for these reasons:

its entity-based modelling approach does not pre-impose the notion of data requirements or inter-relationships between data elements in the way that a record-based modelling approach does;
in the RDF data community it is assumed that different vocabularies of terms (classes and properties for describing aspects of a resource) and concepts (providing the means to classify resources) will be developed in such a way that someone can mix and match terms from relevant vocabularies to describe all the entities that they care about; and
as it is assumed that there will be more than one relevant vocabulary it has been accepted that there will be related terms in separate vocabularies, and so the RDF schema that describe these vocabularies should also describe these relationships.

JDX was designed in the knowledge that it overlaps with schema.org. For example JDX deals with providing descriptions of organizations (who offer jobs), and with things that have names and so does schema.org. It is not necessary for JDX to define its own class of Organizations or property of name, it simply uses the class and property defined by schema.org. That means that any data that conforms to the JDX RDF schema automatically has some data that conforms with schema.org. No need to extract and transform RDF data before loading it when the modelling approach and vocabularies used are the same in the first place.

Sometimes the match in terminology isn’t so good. At some point in the future we might, for example, be prepared to say that everything JDX calls a JobPosting is something that schema.org calls a JobPosting and vice versa. In this case we could add to the JDX schema a declaration that these are equivalent classes. In other cases we might say that some class of things in JDX form a subset of what schema.org has grouped as a class, in which case we could add to the JDX schema a declaration that the JDX class is a subclass of the schema.org class. Similar declarations can be made about properties.

by querying the data provided about things along with information about relationships between the data terms used we can achieve interoperability across data provided in different data standards

The reason why this is useful is that RDF schema are written in RDF and RDF data includes links to the definitions of the terms in the schema, so data about jobs and organizations and all the other entities described with JDX can be in a data store linked to the definitions of the terms used to describe them. These definitions can link to other definitions of related terms all accessible for querying. This is linked data at the schema level. For a long time we have referred to this network of data along with definitions, which were seen as sprawling across the internet, as the Semantic Web, but more recently it has been found to be useful for datastores to be more focused, and the result of data about a domain along with the schema for those data is now commonly known as a knowledge graph. What matters is the consequence that by querying the data provided about things along with information about relationships between the data terms used we can achieve interoperability across data provided in different data standards. If a query system knows that some data relates to what JDX calls a JobPosting (because the data links to the JDX schema), and that everything JDX calls a JobPosting schema.org also calls a JobPosting (let’s say this is declared in the schema) then when asked about schema.org JobPostings the query system knows it can return information about JDX JobPostings. RDF data management systems do this routinely and, for the end user, transparently.

That’s lovely if your data is in RDF; what if it is not? Most system-to-system interoperability standards don’t use RDF. This is the problem taken on by the Data Ecosystem Schema Mapper (DESM) Tool. The approach it takes is to create local RDF schema describing the classes, properties and classifications used in these standards. The local RDF schema can assert equivalences between the RDF terms corresponding to each standard, or from each standard to an appropriate formal RDF vocabulary such as JDX. Data can then be extracted from the record formats used and expressed as RDF using technologies such as the RDF Mapping Language (RML). This would allow us to build knowledge graphs that draw on data provided in existing systems, and query them without knowing what format or standard the data was originally in. For example, an employer could publish data in JSON using HR Open Standards’ Recruiting Standard. This data could be translated to the RDF representation of the standard created with the DESM Tool. Relationships expressed in the schema for the RDF representation would allow mapping of some or all of the data to JDX JobSchema+, schema.org JobPosting and other relevant standards. (The other standards may cover only part of the data, for example mapping skills requirements to standards used for competencies as learning objectives in the education domain.) This provides a route to translating data between standards that cover the same ground, and also provides data that can link to other domains.

Acknowledgements

Stuart Sutton, of Sutton & Associates, led the creation of the JDX JobSchema+ and originated many of the ideas described in this blog post.

Many thanks to people who commented on drafts of this post, including Stuart Sutton, Danielle Saunders, Jeanne Kitchens, Joshua Westfall, Kim Bartkus. Any errors remaining are my fault.

Writing this post was part of work funded by the U.S. Chamber of Commerce Foundation.

The post JDX: a schema for Job Data Exchange appeared first on Sharing and learning.

Dublin Core Tabular Application Profile⤴

from Phil Barker @ Sharing and learning

Over the last year or so I have been contributing to a Dublin Core Metadata Initiative Interest Group, lead by Karen Coyle, that has been drafting recommendations on how to create and document application profiles in simple a tabular format. At the end of last year we had a webinar presenting the work so far, which now available on YouTube. For me the really useful thing about this approach, which we are calling DC TAP, is that it is a human-friendly approach that also lends itself to machine processing—at the end of the webinar recording Tom Baker demos a python script that will take a tabular application profile exported in CSV format and turn it into ShEx that can be used to validate data.

Application Profiles

“data elements drawn from one or more namespaces, combined together by implementors, and optimised for a particular local application” –Heery and Patel

Application profiles are a powerful concept, central to how RDF can be used to create new schemas without creating entirely new standards. The idea was presented in “Application Profiles: Mixing and Matching Metadata Schemas” by Rachel Heery and Manjula Patel, where it was defined as: “schemas which consist of data elements drawn from one or more namespaces, combined together by implementors, and optimised for a particular local application”. Twenty years on and the only part of that definition that I would quibble with is that “local” underplays how widely useful an application profile can be when shared. A couple of examples might illustrate this.

The Data Catalog Vocabulary (DCAT) is a “vocabulary designed to facilitate interoperability between data catalogs published on the Web”. It draws on terms from external vocabularies like Dublin Core and FOAF; where these aren’t sufficient it mints new terms, and it mixes these terms according to a domain model relevant to data sets. Thus DCAT can be seen as an application profile of several pre-existing vocabularies supplemented with few new terms and a model of how to use those terms. Recently, when I was asked about describing learning resources about data management, I suggested adding LRMI into this mix and so create a schema using terms that are mostly already familiar in that domain.—No need for the implementors to move to a new specification. (Hence the use of DCAT at the end of my previous post on harmonizing LOM metadata with RDF.)

Bioschemas “aim to improve the Findability on the Web of life sciences resouces”. They take a similar approach, but with the starting point of schema.org. Unlike Dublin Core and FOAF, schema.org is a huge and ever-growing vocabulary, and the main issue in using it is one of selection, i.e. cutting it down to size, and explanation, i.e. explaining how you want to use what has been selected. On their website you can see their draft recommendation for how they would describe Courses; note the tabular format, how they tweak the term descriptions to be understandable for their community, how they add requirements such as which properties must be used and whether a property can be repeated, also they place constraints on the values allowable for some properties, e.g. that the value should be drawn from a specific concept scheme. This general approach will also be familar to anyone who has looked at Google’s guidlines on use of schema.org for various search products it offers, for example recipes.

Tabular Application Profiles

Select, Explain, Constrain, Share

One of my colleagues in the DC TAP interest group, Nishad talks about “profiling” “contraining” and “explaining” as the key aspects of an Application Profile. Application Profiles allow you to select which terms you want to use, explain how to use them, place constraints on their use (i.e. narrow down what is allowed in the definition of the vocabulary) and share them with others. Existing standards, notably ShEx and SHACL overlap with this approach, but the aim of the DC TAP group has been to create a simple format congruent with the approach that is often taken by the people who build application profiles. Hence the tabular approach rather than an RDF specification, which I feel will provide a format that fosters collaboration around these profiles. The simplest allowable TAP is a single column list of the properties that have been selected. The most complex allows you to specify which properties are used to describe which types of entity, how they are related, what formats should be used for values, which gets more technical but the tabular approach to preseting this helps keep it manageable. Importantly, the output is actionable, e.g. can be transformed to ShEx and SHACL to allow data selection and validation services, or can be parsed to automatically create data entry forms, etc.

There’s plenty of further expanation in the video (embedded above) and in the draft specification and draft primer, but I’ll end with screen shots of a simple example: a schema for describing a course. The vocabulary used is schema.org, so the central entities are CourseInstances (offerings of a course) with specific start and end dates, at a virtual or physical location, about some concept, and with a person as the instructor. First the circles-and-arrows diagram, then the corresponding TAP.

The post Dublin Core Tabular Application Profile appeared first on Sharing and learning.