Author Archives: @cullaloe | Tech, tales and imagery

Video including closed captions, method 2⤴

from @ @cullaloe | Tech, tales and imagery

Further to this recent post on capturing, processing and captioning video, my colleague, Dr. Audrey Cameron, advised me to try YouTube for capturing the .srt captions file more quickly. I am thankful to her for this, because although I was aware of YouTube’s rapidly improving automatic captioning (I use it myself when I watch with sound off, for example), I didn’t know that the .srt file can be downloaded. Here’s a revised approach I am using today.

Capturing the video and audio

For capturing an old-fashioned lecture-style talk using KeyNote, I use the facility to record a presentation (from the Play menu). The presentation can be exported as a .m4v video file with sound. At the same time as recording the presentation in KeyNote, I also record myself on my Fuji X-T2 camera and capture a high quality audio track separately on a Zoom H-1 recorder. A “pro” tip to is to clap just before you start presenting – it leaves a nice spike in the audio waveforms, making it easy to line up the separate tracks. You can also pretend to be Steven Spielberg or Alfred Hitchcock, according to your genre, by saying, “Action!” at the same time.

Processing

Import the video and audio tracks into iMovie, align them using the spike from your clap, and check that the audio is the same length as in the video track of the speaker. I found that for longer videos, over 15 minutes, they can be different. This difference produces an echo effect, eventually separating the video from the soundtrack, like a Swedish movie. Adjusting the audio to align properly can easily be done in iMovie using the speed adjustment. Once the clips are aligned, you can turn the audio level of the video clips down to zero, so only the high quality track remains.

The next thing I do is to change the video track of the presenter to “Picture in picture”, so viewers can see me presenting within the slides: I think this is a bit of a substitute for one of the features I miss from live presenting, which is managing the attention of the viewer. I normally do this by blanking the display, which has the effect of moving the eyes in the room from the screen to my face – a powerful way to add contrast to your talk. This “mini me” within the slides can be faded in or out, according to what you want the students to focus on at any point. Other effects are possible, like switching to embedding the presentation within the presenter video.

The finished project can be exported via the Share menu to a .mp4 file.

Getting the transcript

The video can be uploaded to YouTube now: you’ll need a verified account to upload clips longer than 15 minutes, which means giving Google your phone number. I baulked at this at first, but expedient is the slayer of principle, and in this case, privacy. Make sure you click “Private” when saving the clip. These lectures are not for public consumption.

After a while – maybe 30 minutes, depending on the length of your video – the automatically-generated captions file can be edited using a really nice editing interface in YouTube Studio designed for the purpose. You will need to add punctuation and if you wish, add comments to your own commentary. Once that has been done, the .srt file is ready to download.

Media Hopper Create

Your video can now be used within your local VLE, in my case, Blackboard Learn, by uploading via the media manager, Media Hopper. Once uploaded, you can then add closed captions by uploading the .srt file alongside it. Students then have a choice whether to access captions within the video sequences or not.

Another Hitch

I was about to post this, all smug, like, as I uploaded the latest video made with this method, when I hit a “file too large” error when uploading to Media Hopper. The video I had made was just short of 18 minutes and had a file size of 1.2GB. Now, mp4 is an efficient container format so I maybe made too many “best quality” choices in making the video: high definition 1080p for the presenter, same for the KeyNote. Rather than go back and do it all again, I resorted to ffmpeg to make me a reduced bitrate version. I thought halving the bitrate might produce a file half the size.

$ ffmpeg -i mybigvideo.mp4 	# find out what the current bitrate is..
...
  Duration: 00:17:28.55, start: 0.000000, bitrate: 9978 kb/s
...
$ ffmpeg -i mybigvideo.mp4 -b 4489k mysmallervideo.mp4

This made (after thrashing my 5-year old MacBook Air for about 25 minutes) a file – as hoped for – half the size (673 MB).

Deployment to a website

To use the video and captions file together within a webpage is straightfroward, except that the captions need to be in a different format. This format is Web Video Text Tracks (VTT), and is easily obtained using ffmpeg:

$ ffmpeg -i srtfile.srt subtitlefile.vtt

The web page needs the following code (adapted to your own file paths, obviously):

<p>  
<video width="640" height="360" controls="controls">
	<source src="https://www.learn.ed.ac.uk/path-to-video.mp4" type="video/mp4">  
	<track src="https://www.learn.ed.ac.uk/path-to-vtt-file.vtt" kind="captions" srclang="en" label="English" default>
	Your browser does not support the video tag.
</video>
</p>

Conclusion

Video production for ‘digital first’ teaching strategies needed in response to COVID-19 measures or similar, is a non-trivial task. Including closed captions is an additional time multiplier. Personally, I don’t like asynchronous teaching at all: it misses so many important aspects of good pedagogy, aspects which are easily ignored by administrators of education in the pursuit of apparent economies or easy fixes. I am at ease, however, with an established workflow.

I am thankful to my colleague, Audrey, for her patience and support in helping me get to this solution. We both have a lot of videos to make, and now it will not take me as long as it might have done.

IOP CLPL: Smartphone practicals⤴

from @ @cullaloe | Tech, tales and imagery

The IoP in Scotland is putting on a rich catalogue of online CLPL for the community of physics teachers, and presented by colleagues from that same community. I accessed one of these after the event, because it had been recorded and published by Drew Burrett on YouTube.

This session was hosted by Stuart Farmer and Jenny Hargreaves, and presented by Murat Gullen and Martyn Crawshaw, who gave us a practical introduction to using tools for data capture and analysis using the suite of sensors on most modern phones. They presented a brief rationale, acknowledging that although not all children have a smart phone, most have access to one. The scope (pun intended) for teachers to offer practical sessions using their own equipment was also underlined.

The first tool discussed was PhyPhox which not only can access all of the phone’s sensors, it has a number of built-in activities and tools to make use of the data captured. These can be exported in several formats, or posted to Dropbox for later discussion. What I hadn’t realised is that there is a built-in web server accessed through the “triple dots” at the top right of the phone screen which enables display and control of the app from any nearby browser, provided that the computer is on the same network (it uses a 192.168.x.x IP address).

Screenshots of Phyphox on iPhone and Browser (uses the language of the phone)

Martyn started by talking about Vernier video tracking and analysis software, similar to the java Tracker program. He went on to demonstrate Pasco’s SparkVue app, what Martyn called the “twenty-first century version” of Data Studio found in many schools. It allows connection to onboard sensors and also Pasco equipment in the lab.

It was made clear that teachers should take care not to assume that their pupils are tech-savvy enough to know how to use all of the tools and interfaces without support and guidance.

“Don’t lose the learning!” – Martyn Crawshaw

Excellent stuff, as ever, with a lively Q & A at the end. I’m going to share this with my students.

Video capture and production including closed captions⤴

from @ @cullaloe | Tech, tales and imagery

Introduction

I am required to produce video resources for my students who are coming to the University very soon, either in person or digitally: our teaching under the COVID adjustments is “digital first”. We are also particularly keen to support those students who might require captions on their videos. This isn’t just those who might be hearing or visually impaired, it’s all students who might like the ability sometimes to have the extra clarity provided by words on the screen that reflect the words spoken by the presenter.

Here’s my take on a workflow model to make this work well. There are existing facilities to do this semi-automatically – uploaded videos can have an automated transcription generated but this takes a lot of time, and requires the creator to go back on subsequent days to hand-edit any errors or make any other adjustments.

I like to do a task and complete it: I like to get it right and take my time over that. Once it’s done, I like to move on to the next set of tasks. To that end, here is a way I have found of creating video, with quality captioning for those who need it, and the ability to switch it off for those who find it a distraction.

Contents

Subtitles vs. captions

Subtitles are embedded in DVD movies and the like for languages other than that on the audio. Multiple subtitle tracks can be embedded within a title, or switched off if the audio track is in the native language of the speaker. Captions, on the other hand, are a transcription of dialogue in the video. Closed captions are distinguished from open captions by being able to be switched off if required. Open captions are often embedded in the video and cannot be turned off. My intention in this workflow is to provide closed captions.

Capturing the video and audio

I captured my video for the proof-of-concept using a Fuji X-T2 APS-C digital mirrorless camera which takes 1080p (1920 x 1080) video. Although the camera records stereo audio, I prefer to capture audio separately using a Zoom H-1 recorder. The quality is much better, not least because the mic is with the speaker, not the camera.

Processing

I import the video and audio tracks into iMovie, trim out the top and tail and make other edits, and remove the embedded audio track captured by the camera. This is replaced with the Zoom audio, which can be a bit of a fiddle to align well with the video. The audio waveform can help here but that depends on the recording environment. Depending on the exposure settings on the camera, you might want to “enhance” the video in iMovie for a contrastier image. Once you have added any title sequences (or credits) and transitions, export the finished video (via the “share” menu) as an mp4 file.

Getting the transcript

Open the mp4 file in Quicktime and export as an audio file, the default m4a format is OK. This file can be uploaded to a blank Word document in Office 365 via the Dictate drop-down menu (select Transcribe). This will do a pretty decent job of turning your dialogue into text.

Save the .docx file and convert it to a plain text format - I prefer markdown, although this isn’t necessary provided you can finish up with a plain text file. I do the conversion using pandoc:

$ pandoc transcript.docx -o transcript.md

This file can now be edited, correcting any errors in the transcription and chunking the dialogue to make it show at the right time during the video. This is easily done with the video window open on the desktop next to your editor. You should add section markers and timestamps for subtitles as you go through the video. A minimal example:

1
00:00:01,00 --> 00:00:01,30
Welcome to my interesting video.

Finally, you will need this transcript in the correct format for embedding into the video file. This is a simple matter of changing the filetype to .srt (which is a “SubRip Subtitle” file).

Add Closed Captions to the video file

The ffmpeg tool is best for this. This tool (and others you might need) can be installed simply using brew if it isn’t already on your system. I don’t propose to detail how to do this, but in essence, it’s a matter of typing$ brew install ffmpeg in a terminal window. Once you have it, add the captions:

$ ffmpeg -i videofile.mp4 -i transcriptfile.srt -c:v copy -c:a copy  -c:s mov_text -metadata:s:s:0 language=eng output.mp4

The program accepts input files, which are you video and the srt file containing the captions and timings. Video and audio are merely copied to the output file.

Finished, part one

The video file you have just created is now shareable: users can play it on their machines and opt to switch captioning on or off if they wish. Their computer may choose to control this behaviour automatically if local settings allow it.

I need to distribute this video using resources within the VLE (virtual learning environment, in my case, Blackboard Learn and Media Hopper. This is where it gets sticky.

Media Hopper Create

It’s easy to upload a video to the Blackboard VLE by clicking on Media Hopper Create in the Tools menu. This is very nice but this process strips out the captions track. Embedding the uploaded video offer no CC option to viewers and no captions are visible. This is clearly a fault in the Media Hopper Create system.

You can ask for subtitles to be created for the uploaded video but this is an automated and low-quality service that isn’t really any good. It creates, ironically, a CC track that is inferior to the one included in the uploaded file.

A workaround

I have found a way of getting around this difficulty: within Learn, add a new item. This is effectively a webpage, and using the editing tools, you can upload two files. The first is the mp4 video file (it is not necessary for this file to have the embedded captions track).

The second file contains the captions and timing information in our srt file, but needs to be in a different format. This format is Web Video Text Tracks (VTT), and is easily obtained using ffmpeg:

$ ffmpeg -i srtfile.srt subtitlefile.vtt

Having uploaded these two files to the Learn Item, it is necessary to edit the HTML using the built-in HTML editor (click the double chevron at the right end of the tool bar to reveal it). The item source should be edited to contain a <video> tag:

<p>  
<video width="640" height="360" controls="controls">
	<source src="https://www.learn.ed.ac.uk/path-to-video" type="video/mp4">  
	<track src="https://www.learn.ed.ac.uk/path-to-vtt-file" kind="captions" srclang="en" label="English" default>
	Your browser does not support the video tag.
</video>
</p>

The path to video and the vtt files is available in the links to them put thee by Learn when you uploaded the files. It is not necessary to keep these links.

Finished, part two

This is a workaround but we now have the facility to make and upload good quality video with closed captioning that can be viewed by students within their Learn course.

Merge join data files on 2 columns with python⤴

from @ @cullaloe | Tech, tales and imagery

This was posted on a forum:

I have two enormous data sets - 2 million rows in each one. I have them in ASCII format. Each set has three columns. The first two columns are identical for both sets - essentially, coordinates. The third row in each set gives the temperature at that location for two different substances.

I am trying to find a way to create a single table with 4 columns, the first two being the coordinates and the third and fourth being the different temperatures.

Excel gives up after 1,000,000 rows.

Can anyone suggest a (free) tool that can do this - and then preferably allow for some analysis - plotting temp 1 against temp 2, for instance.

I thought about this on and off during one of those challenging COVID days, and finally sat down in the evening and knocked up a quick solution to the first part of the problem, using python.

file 1.csv, 2.csv contain data:

$ cat 1.csv 
"x","y","t1"
42,35,122
39,44,242
12,43,188

$ cat 2.csv 
"x","y","t2"
53,22,192
39,44,122
22,56,238

Launching python 31

$ python3
Python 3.6.2 (v3.6.2:5fd33b5926, Jul 16 2017, 20:11:06) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> csv1 = pd.read_csv("1.csv")
>>> csv2 = pd.read_csv("2.csv”)

(check it looks ok)

>>> csv1.head()
    x   y   t1
0  42  35  122
1  39  44  242
2  12  43  188
>>> csv2.head()
    x   y   t2
0  53  22  192
1  39  44  122
2  22  56  238

Make an outer join of these two tables on the two coordinate columns:

>>> merged_data = csv1.merge(csv2,on=["x","y"],how='outer’)

>>> merged_data.head()
    x   y     t1     t2
0  42  35  122.0    NaN
1  39  44  242.0  122.0
2  12  43  188.0    NaN
3  53  22    NaN  192.0
4  22  56    NaN  238.0

“NaN” is not your Nan, it’s just “not a number”, or “no data here”.

>>> merged_data.to_csv("out.csv",index=False)

… and back in the shell, we can see the result…

$ cat out.csv 
x,y,t1,t2
42,35,122.0,
39,44,242.0,122.0
12,43,188.0,
53,22,,192.0
22,56,,238.0

… your merged file, sir.

It ought to work with a few million rows. I didn’t answer the final part of the OP’s question but it ought to easy enough with this example - a million or more rows, might be a different problem.

Footnotes

  1. I am using python3 because OSX has a legacy version 2 nobody dares touch. On other systems you might want to use just “python” and “pip” in the above examples. You might have to install pandas first, using $ pip3 install pandas

Practice worth sharing seminar⤴

from @ @cullaloe | Tech, tales and imagery

This is one of a number of “sharing good practice” seminars organised by my colleagues at Moray House, and was more of an exploratory talk given by Dr. Laura Colucci-Gray. These are my notes, made during her talk: as much reflection as reporting. Any errors are my responsibility.

Laura reflected on observing her own children learning alone during the “lockdown” of COVID-19 and thought of the dynamic of (on the one hand) being connected, but (on the other) being disconnected at the same time. She observed that the machine that connects us leaves us no environmental cues, compared to the “real world” environment, which has displays, visual clues, artefacts of daily working to attest to the impact of our interactions with other human beings.

She included readings from literature on the work of the teacher, who has a role of “packaging” content for learners and set this in contrast to the context of learning in a social setting. This tension, she characterised as a danger in asynchronous learning contexts. She placed learners in our thinking not so much as individual units but humans undergoing a process of co-operative effort that takes place between binaries like male/female, nature/technology, local/global, and so on.

The traces we leave behind need to be thought about when planning and teaching in an asynchronous space. Laura exemplified this in image-making, experience of physical spaces, and in “utilising modes other than language creates space for different ways of knowing or being” (Liamputtong & Rumbold, 20081). The latter quote is in context of the use of music as making traces as a “soundtrack” to a learning course.

The theme emerged further in Laura’s ideas for using traces to provoke shared experiences, and through these shared experiences, my view is that a stronger learning community is established. She provoked the delegates to think and talk about how we can make use of the technology to create artistically shared experiences; how far to go to enable sharing something of themselves and teachers; what opportunities do we have in putting ourselves in the process of co-learning with our students?

At this point, a rich discussion followed, exploring the possibilities for us as teachers moving forward into a more asynchronous mode of practice. For me, this was an outstanding provocation for my thinking, as I set about more detailed preparation for new students arriving (virtually, mostly) very soon.

My own comment related to Lackovic’s inquiry graphics2 that we have worked with in interpretation and creation of imagery for understanding. I went on to ask how we develop the skills in our students to be able to communicate and express themselves using drawing and music, as much as we do with academic literacy now.

The final part of the discussion considered the opportunity and challenge in enabling safely the participation (or non-participation) of students in activities designed to “share something of themselves”. This was powerful and no less provocative to our practice and a memorable conclusion to an excellent session.

Notes

  1. Liamputtong, P. (2008) Knowing differently : arts-based and collaborative research methods, New York, Nova Science Publishers. 

  2. See, for example, Lackovic, N. and Crook, C. (2012) ‘Designs for learning, image-based concept inquiry, a DBR research project’, in Designs for Learning 2012, Designs for Learning, Copenhagen, Denmark, Aalborg Universitet København, pp. 47–48. 

Nextcloud calendar embedding⤴

from @ @cullaloe | Tech, tales and imagery

I needed to embed a NextCloud calendar within another website. Next cloud offers an embed code block which looks like this:

<iframe width="400" height="215" src="https://your.tld/index.php/apps/calendar/embed/50m3rAnD0mC0d3z"></iframe>

Because NextCloud sets some fairly intelligent security defaults, this results in the browser throwing a “refused to connect” error unless the site you embed the calendar iframe on is on the same domain as your NextCloud instance. See the header image for what this looks like.

The solution lies in allowing your target web server to connect to your NextCloud server. This is not done in NginX/Apache settings because these will be overridden by NextCloud anyway. Instead, edit the ContentSecurityPolicy.php file for NextCloud. This file is located in the /lib/public/AppFramework/Http/ folder of your NextCloud installation.

/** @var array Domains from which iframes can be loaded */
	protected $allowedFrameDomains = [
	'https://*.yourdomain.tld',
    ];
    
/** @var array Domains which can embed this Nextcloud instance */
	protected $allowedFrameAncestors = [
		'\'self\'',
    	'https://*.yourdomain.tld',
    ];

With the above edits (no server restart required), the calendar will embed quite happily. There is no need to implement the X-Frame-Options HTTP header documented elsewhere as this is obsolete. The Content-Security-Policy header delivered by the above fix from within NextCloud is enough.

Datacamp course – intermediate R⤴

from @ @cullaloe | Tech, tales and imagery

Continuing my journey into R, the next course in the R programming track at DataCamp is Intermediate R. This course is presented by Filip Schouwenaars. It teaches language syntax and programming conventions, building on the last course.

Conditionals and Control Flow

Relational operators

This section begins with a talk-through the main relational operators in R, with simple examples, followed by exercises in the virtual lab.

> TRUE == TRUE			# Equality
[1] TRUE
> 'oranges' != 'apples'	# Inequality
[1] TRUE
> 'oranges' > 'apples'	# Strings compare alphabetically
[1] TRUE
> 'oranges' < 'apples'
[1] FALSE
> vec <- c('apples', 'bananas', 'dragon fruit', 'tomato')
> vec > 'oranges'		# Works on vectors (and matrices)
[1] FALSE FALSE FALSE  TRUE

TRUE coerces to the value 1, FALSE, 0. So truth is greater!

Logical operators

Syntax for these familiar operators is &, | and !, for logical AND, OR and NOT, respectively. They have high precedence and therefore do not need brackets around expressions:

> 4 > 3 & 8 <= 9
[1] TRUE

Logical operators may be used on matrices and vectors:

> !c(TRUE, FALSE, 1 > 0)
[1] FALSE  TRUE FALSE

Note that double-signed operators like && work only on the first element of a vector.

Conditional statements

Again, familiar syntax here, with the conditional test in brackets; code blocks in curly braces; and two statement words, if and else:

x <- 0
if (x < 0) { 
    print ('x is negative')
} else if (x == 0) { 
  print ('x is zero') 
} else { 
    print ('x is positive') 
}

Notice that the else and else if statements come on the same line as the closign curly brace of the associated if statement. Once a conditional test evaluates TRUE, the corresponding code block is executed and the remaining code within the if control structure is ignored. Conditional statements may be nested.

Evaluation and next steps

There is a greater teacher presence in this course than the previous, through the use of video presentations to support the hands-on interactive labs.

Thus far into the R Programming Track with Datacamp, I have stopped because I have hit an unexpected paywall. Continuing requires a commitment of at least $25 per month, which is good value if I were continuing with courses several hours per day but not appropriate for my current ad-hoc engagement. The day job takes priority, which means all of the available time mostly. I’ll be switching to other resources from now, probably starting with R for Data Science1, or at least the online version.

References

  1. Grolemund, G. and Wickham, H. (2016) R for Data Science, O’Reilly Media. 

Datacamp course – introduction to R⤴

from @ @cullaloe | Tech, tales and imagery

Having abandoned the data visualisation course run by Edinburgh University, and wanting to gain some further competence in R, I took the DataCamp “Introduction to R” course. This course is written by Jonathan Cornelissen, one of the founders of DataCamp and a man with seriously good credentials in R.

Basics

Assignment and operators

a <- 4		# assignment 3 ways
4 -> a
a = 4

1 + 2		# mathematical operators
4 - 3 
6 * 5
(7 + 9) / 2 
8^2		# exponentiation
10 %% 4		# modulo

x < y		# less than
a > c		# greater than
a <= b 
j >= k 
one == two	# equal to
up != down	# not equal to

Data types

12.5 / 2.5	# numerics
7 + 123		# integers are also numerics
7 = 3		# Boolean (TRUE or FALSE) are logicals
"Hello world"	# characters

class(x)	# what data type is x?

Vectors

A vector is a one-dimensional array (think of a row in a spreadsheet). In research, this is a single observation.

# using the combine function to create a vector
a_numeric_vector <- c(1, 2, 3, 4, 5)

# vectors can have column names
names(a_numeric_vector) <- c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday")

# printing the vector outputs the element names:
> a_numeric_vector
   Monday   Tuesday Wednesday  Thursday    Friday 
        1         2         3         4         5
> 

# using a vector to hold the column names
days_of_week <- c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday")
names(week-values) <- days_of_week

You can do some quick and easy arithmetic with vectors.

low_nums <- c(1, 2, 3, 4, 5)
hi_nums <- c(6, 7, 8, 9, 10)

total_nums <- low_nums + hi_nums

> total_nums
[1]  7  9 11 13 15

sum(low_nums) 	# adds up the elements in the vector
mean(low_nums)	# average of elements in the vector

low_nums[3]	# print the third low number (note 1-index)
hi_num[c(2:4)] 	# just get the middle values

The selection of elements can be conditional using boolean values in another vector.

> c(49, 50, 51) > 50
[1] FALSE FALSE TRUE

> nums <- c(1:99)	# vector of the first 99 integers
> fives <- nums %% 5
> nums[fives == 0]	# all of those divisible by 5
 [1]  5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

In the last example above, fives == 0 is a vector of boolean values. Used as a selector in the nums vector, only the TRUE elements are selected.

Matrices

A matrix in R is a collection of elements, all of the same data type, arranged in 2 dimensions of rows and columns.

> # A matrix that contain the numbers 1 up to 9 in 3 rows
> matrix(1:9, byrow = TRUE, nrow = 3)
     [,1] [,2] [,3]
[1,]    1    2    3
[2,]    4    5    6
[3,]    7    8    9
> 

The access indicators are shown in the row labels and column headers above. So, element [2,3] of the matrix contains the value 6. The first row of my_matrix is the vector my_matrix[1,]. Row and column names can be set for matrices, as they can be for vectors. This can be done by calling rownames() and colnames(), or at the time the matrix is set up.

# Construct star_wars_matrix
box_office <- c(460.998, 314.4, 290.475, 247.900, 309.306, 165.8)
star_wars_matrix <- matrix(box_office, nrow = 3, byrow = TRUE,
                           dimnames = list(c("A New Hope", "The Empire Strikes Back", "Return of the Jedi"), 
                                           c("US", "non-US")))

The function cbind() binds columns to an existing matrix. rbind() does the same thing for adding row vectors to a matrix. rowSums() and colSums() do what they sound like - making new vectors ready to be bound into the source matrix if required.

Arithmetic operators work element-wise on matrices.

Factors

A factor is a data type used to store categorical variables. These are discrete variable which can only take a finite number of values (cf. continuous variables which can have any of an infinite set of values, like real numbers). R can make a vector of the categories from a vector of categorical values:

> birthdates <- c(12,4,13,23,31,16,1,9,12,4,8,24,27,25,24,25)
> birthdates
 [1] 12  4 13 23 31 16  1  9 12  4  8 24 27 25 24 25
> bd_factors <- factor(birthdates)
> bd_factors
 [1] 12 4  13 23 31 16 1  9  12 4  8  24 27 25 24 25
Levels: 1 4 8 9 12 13 16 23 24 25 27 31
> 

Such variables are nominal or ordinal according to whether they are just names, or if they can be ranked in some meaningful way. Ordinal factors are created with additional parameters, e.g., order = TRUE and levels = c("low", "high") and can be compared easily.

Data Frames

A data frame has the variables of a data set as columns and the observations as rows. A quick peek at the structure of a data frame is provided by head() and tail() functions, e.g.:

> head(mtcars)
                   mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
>

mtcars is one of the many data sets built into R. A list of them is obtained by calling data(). str() provides a look at the structure of a data set:

> str(mtcars)
'data.frame':	32 obs. of  11 variables:
 $ mpg : num  21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
 $ cyl : num  6 6 4 6 8 6 8 4 4 6 ...
 $ disp: num  160 160 108 258 360 ...
 $ hp  : num  110 110 93 110 175 105 245 62 95 123 ...
 $ drat: num  3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
 $ wt  : num  2.62 2.88 2.32 3.21 3.44 ...
 $ qsec: num  16.5 17 18.6 19.4 17 ...
 $ vs  : num  0 0 1 1 0 1 0 1 1 1 ...
 $ am  : num  1 1 1 0 0 0 0 0 0 0 ...
 $ gear: num  4 4 4 3 3 3 3 4 4 4 ...
 $ carb: num  4 4 1 1 2 1 4 2 2 4 ...
> 

Columns of a data frame are added one column vector at a time as a list of parameters in the function call data.frame(). Selecting a data point from row 32, column 2 is a matter of calling df_bears[32,2]. Note the order - observation (row) first, then variable (column). A whole observation (e.g. the tenth) is obtained by df_bears[10,]. The first 4 data points from the paw_size column are df_bears[1:4,"paw_size"]. The whole column vector is df_bears$paw_size (notice the dollar sign notation). Subsets can be made calling subset(df_bears, paw_size < 4). Sorting can be achieved by making a vector of the data frame order, based upon the columns you are interested in:

> a <- order(df_bears$claw_size)
> df_bears[a,]
> 
> mtcars[order(mtcars$disp),]
>                      mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Toyota Corolla      33.9   4  71.1  65 4.22 1.835 19.90  1  1    4    1
Honda Civic         30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
Fiat 128            32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
Fiat X1-9           27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1
Lotus Europa        30.4   4  95.1 113 3.77 1.513 16.90  1  1    5    2
Datsun 710          22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
...

Lists

Lists can contain arbitrary data and data types. They are constructed by calling list() with optional names for each component, e.g. list(top_dogs = df_dogs[1:10,], top_cats = df_cats[1:10,]).

Evaluation and next steps

I’ve found this introduction differently paced to the earlier introduction course run by the university. Because there is no instructor, attention has been paid to very small details: every aspect of the course works because it is programmatic. Learners have to take the right (small) steps to complete the exercises successfully. Errors are picked up and RTFQ-type prompts are given. This was less challenging than the earlier, demonstrator-led course but I completed this one, instead of bailing out feeling frustrated and weak. I also learned considerably more that is useful and earned a more secure foundation for further study.

I am working with RStudio on a daily basis now as I am producing documentation and course materials with Bookdown. My intention is to further develop competence with R and R-markdown.

OneNote Teacher Academy⤴

from @ @cullaloe | Tech, tales and imagery

The Microsoft OneNote Teacher Academy is a learning path comprising four short courses introducing the use of OneNote in teaching for lesson planning, assessment and activities for learning:

  • Getting Started with OneNote
  • OneNote Class Notebook: A teacher’s all-in-one notebook for students
  • OneNote Staff Notebook: Tools for staff collaboration
  • Independent learning with math tools in OneNote

These are my notes taken as I followed the learning path, including the reflection questions and my responses.

Contents

Getting started with OneNote

The course makes use of lock-step interactive videos which step you through mouse clicks to achieve the various functions in OneNote, which at first I thought was patronising, but which actually gets you used to the interface quickly. If you’re following and feel the same thing, stick with it.

Module 1: Getting started with OneNote for Windows 10

So, the introduction tells us that OneNote is a way of “staying organised, quickly capturing ideas, and accessing information across devices”. It’s a digital notebook that seems to rely on Windows 10 - the intro video seems to assume that all participants are using that OS and steps through “how to use Windows”. The OS is different on my MacBook, so the introductory video doesn’t work well at the start, but it gets better once you start using the OneNote app.

Reflection

What aspects of organizing words does OneNote mimic? - It’s like a real notebook and helps you make little documents or pages of notes.

What are some ways you might organize your sections and pages? - into chapters or topics, maybe. Gather those into project notebooks.

What topics might be an easy place to start a notebook or section? - Course structure, syllabus, one note per week, maybe.

Module 2: Getting to know OneNote for Windows 10

OneNote is organised into pages, which can be organised in sections like the tabs in a FiloFax. It has a built-in full-text search function which I think is a very good thing for students to have - it’s one of the major reasons I turned from Learn to Moodle 3 years ago.

Items like text boxes in the pages of OneNote are drag-and-droppable, a bit like the Apple Pages app, and allow a great deal of flexibility over how the pages look. Each page is like “a digital canvas” that can accept media, images and ink (from a stylus), or embed things like YouTube videos. Items can be hyperlinked.

Audio or video feedback can be inserted into pages on the fly.

User help in the interface is available through the “lightbulb” icon, aka “Tell me”.

Reflection

What are the benefits of being able to start writing or typing anywhere on the page? - you can be creative with the flow and functionality of your resources and activities.

What are the benefits of being able to insert pictures and media anywhere on the page? - same again, creativity in making logical storyboards for learning.

Module 3: Getting the most out of OneNote for Windows 10

This module exemplified the use of digital ink, (re)organising materials and sharing work in a team using OneNote. A number of workarounds in the interface were also exercised in here, such as when you need to right-click, or use a non-obvious menu to perform aesthetic changes like page colour. There are neat functions built in for tablet users in particular, such as “ink to shape” and “ink to text” that convert your scruffy notes and scribbles into something somebody else can read. The glory of form over function!

Inserting video links will embed the video itself.

One very interesting feature is the researcher tool, which is used for pulling together web sources and their citations. I’m not sure it’s going to lead to good academic writing, but I’ll try it out to see how it works.

Reflection

How much time do teachers spend looking for files in old email threads? - none at all. Not this one, anyway, I’m better organised than that. I delete almost everything once I’ve detached the information I need.

How does OneNote’s organization save teachers time? - I’m not convinced it does, yet. I think that quick re-organisation of pages and sections is what is being referred to in this question.

What is the benefit to being able to access information from all of your devices? - you can drop your device off a bridge and not lose any information.

Module 4: Digital Ink

This section described the time-linked digital ink feature of OneNote which allows you to step forward and back through the sequence of pen strokes. A teacher can do this in a student’s notes, too, to enable them to replay, video-like, the thinking of the learner as they constructed their responses. A blog post advertises the features in the recent redesign of OneNote and it does look impressive in the use case shown in the promotional video. I particularly liked the attention that has been paid to achieving consistency across different platforms and devices:

OneNote users often use a range of devices. Having a cohesive user experience across all screens makes it simple for users to jump from one device to the next.

This also helps promote consistency of experience in a learning community, enabling a common culture of understanding in collaboration.

Reflection

When would you encourage students to use the “playback” feature? - when revising previous work, or when looking at teacher feedback.

Would this be beneficial for a teacher looking at a student’s homework assignment? - of course, to uncover misconceptions.

Do you think teachers would use this as they take notes for a class or meeting? - I need to try it first, but it might enable the making of useful video tutorials for revision, or accessibility, or for students who are unable to participate live.

Trying it out

I went to my own instance of OneNote to try this playback feature out but couldn’t find it at all, which was frustrating. Nor could I find things like “ink to text”, the “tell me” lightbulb, or many of the features this course has described. Perhaps the OSX version of OneNote I am using doesn’t know about the consistency across devices this module is talking about. Further reading reveals that although they are superficially similar, there are substantial differences in the feature sets in OneNote, depending on which device you are running it on. This gives the lie to the earlier content, something I am genuinely disappointed to discover, having made a real effort to suppress my visceral, justified and hard-earned distrust of All Things Microsoft.

OneNote links tend to open not in the application but a browser window - yet another version of OneNote that doesn’t have the features described so far in this course. The ecosystem seems to be significantly varied across platforms.

The very last line of module 5:

This course covers features in the OneNote Windows 10 app. To see which features are available in other OneNote versions, visit this site: The differences in OneNote.

The list isn’t complete or accurate, for example, the replay function I was genuinely excited by, isn’t in the list. Spending more time exploring the OneNote interface created more difficulty. I tried to connect to Teams, where I have set up a team for the new cohort and those remaining from previous years, but with little success, and little help when it didn’t work:

Screenshot on connection

Module 5: A content library brimming with rich resources

The last module in this “getting started” course is a collection of videos or examples of content made by other teachers (“OneNote Ninjas”). I couldn’t see how the content library worked at all, nor how it worked in teaching practice. Some clues were deducible from the end of course quiz - it seems to be a read-only resource space for learners.

Reflection

How might the Content Library be a unique place for students in your class? - in the same way perhaps, as the class cupboard is unique.

What kinds of media might you add to make your Content Library a go-to place for students? - not sure, because I couldn’t get it to work, but I would hope any digital media.

Once your Content Library is complete for one class, would it take much to alter it for next year’s class? - it depends on the class and any changes to the course but in principle, maybe just change all the dates.

The quiz

A very low bar to leap over. For example…

  • OneNote can locate any word on any page in any of your notebooks with the ____ function.

    • Looking
    • Hunt
    • Search
    • Where is it?

Still useful, though, because the quiz added some details that weren’t in the course content.

OneNote Class Notebook: A teacher’s all-in-one notebook for students

The course uses the same approach as the first, with interactive videos.

Module 1: Getting started with a OneNote Class Notebook

The Class Notebook is intended to help teachers deliver content, work collaboratively and provide private feedback to learners. It is described as being like a physical notebook, filing cabinet and whiteboard in one. This is achieved by providing a private notebook for each student; a content library; a teacher-only area; and a space for collaboration with students or staff.

The teacher can look at any of the student notebooks, and provide private feedback or commentary. The collaboration space allows for peer work and assessment. The content library is only writable by the teacher.

Reflection

What types of materials would be best housed in the Content Library versus the Collaboration Space? Think of a few ways to organize your Class Notebook and start by creating one section and giving it a try. - Static materials which do not require editing or annotation by students.

I created new sections within the collaboration space and content library, and even created a new section group, but cannot immediately tell how access is managed in these groups. Further investigation revealed that you cannot do this in the OneNote app: it can be done by logging in to the Microsoft OneNote website and finding Manage notebooks. I found this difficult because the browser version of my Class Notebook and the OneNote version were different (different groups, and sections not in sync). The browser interface offered connection to the notebook, which it did in the browser, but still gave me no insight into who could see my new group. The lightbulb “Tell me what you want to do” offered a very narrow scope (menu items) or a wider search using Bing (what else, when you’re in a corporate prison?) which isn’t available without acknowledging data permissions. I’m stuck in a hole within an application infrastructure which is utterly getting in the way of what I am trying to do.

Module 2: Setting up a OneNote Class Notebook

It says here, “OneNote Class Notebook can assist teachers in providing:

  • Individualized instruction
  • Meaningful feedback
  • Rich multimedia content”

We’re getting a lot of repetition in the videos now, and more patterns emerging. Setting up a OneNote Class Notebook starts not in the app, but on (yet another different) website: Office 365. This can also be done via the MS Teams app: I set up a class notebook for the new cohort Team OK. The dialog for this task sets out the default access permissions for the Collaboration Space; Content Library; and Student Notebooks.

Class notebook setup

Once the class notebook is set up in Teams, you can open it in the OneNote app. Deleting a notebook is a completely different matter: it is ridiculously hard to do this and will take you and hour or so of Googling: different procedures are needed according your platform, Office account setup and which way the wind is blowing. I followed this set of instructions to delete the class notebook I had set up, naïvely, in the OneNote app. That didn’t work at all, because the menu selections on the website were different (there is no “Manage and Delete” option in OneNote today). I have no idea how to delete a NoteBook. I think it can be done through the online access to OneDrive, but I can’t tell if I deleted a link, or an actual item, or whether in fact it was just moved somewhere else. That situation gives me no confidence in this software suite at all. The more I learn about it, the more I dislike it. Still, we’re here now, so let’s see this through…

Reflection

In what ways would OneNote Class Notebook allow you to differentiate your instruction to various students? - by writing different things in their notebooks. By providing different materials for them in the content library.

What is it about Class Notebook that might expedite giving feedback compared to traditional paper assignments? - synchronisation. The student’s notebook is updated as soon as the teacher writes in it.

Module 3: Getting the most out of OneNote Class Notebook

It says here, “OneNote Class Notebook has many features to help teachers:

  • Distribute assignments
  • Write feedback
  • Share with parents - a unique link”

OK, I get it. It’s like Church. Keeping saying the mantras, singing the hymns, re-stating the creed, and eventually we all are saved. The more I am sold to, the less I want to buy. I’m going straight to the reflections before watching the video this time:

Reflection

What are the advantages to verbal feedback for a student? - it seems personal, and can be nuanced in non-verbal ways through tone and gesture. It can be individual, and can be replayed (if a clip is available, which presumably it is) at different speeds for comic effect to help understanding.

What are the advantages to verbal feedback for a teacher? - it might be quicker, especially if you do it in one take. There are some very real disadvantages too, in that it can be replayed in contexts you hadn’t intended.

When might teachers use this feature? - I wouldn’t normally, unless I had a vision impaired student who preferred it, but can see why some might be tempted to try it.

Things just don’t work

I discovered more functionality glitches as I followed the “getting the most…” video. I think these are bugs, rather than poor connectivity between the various interfaces - Teams, the OneNote app, the different websites that manage the spaces, etc. Things just don’t work: it’s clunky and inconsistent in daily usage, resulting in huge time costs for the teacher. For example, in OneNote app, I found “Manage Notebooks” which popped up what looks like a browser container, where I added a teacher only space without errors. This did not add the teacher only space in the Class Notebook in the app, but it was visible in the browser. Browser and app were synching OK, because I could type in one and see the changes with less than a second of latency in the other1.

I like the idea of “distributing” a page (assignment) to all student notebooks at the same time, then being able to review all of their completed edits of their copy of that page (assignment) simply in the interface. I am not sure I can rely on that without testing it first, however, so am not intending to spend any time on this. The collaboration space idea is implemented in a difficult and weird way also: individual groups of students can be allocated to sections within that space but if a deadline passes, the entire collaboration space must be locked rather than the individual section.

I am increasingly coming to the conclusion that BlackBoard Learn might be my best option for Physics PGDE in the coming session. This is a real pity because of Learn’s deficiencies. I wonder if I will go maverick again and launch a Bookdown site as a searchable resource for my students.

Module 4: Immersive Reader and Researcher

This module provides detail on some of the accessibility features built into Class Notebook, including the Immersive Reader, which is a nice idea but again, just does not work reliably. It failed to recognise login credentials when I tried to use it the first time but I was able to get it working once I’d logged in and out a few times of the various interfaces. When it does work, it’s a screen reader - “immersive” meaning that it removes sidebars and ribbons, displaying just the text of the document you’re looking at in a large font, with a moving highlighter that follows the voice.

Researcher is a weak tool that lowers the access bar for students who need help doing basic searches for information. It seems to have been lifted, like other tools in this suite, from the main Office toolset.

Reflection - see above.

Module 5: Making Math Happen in OneNote

This module is not about learning mathematics and is definitely not about “… a virtual math tutor with OneNote”. It is just a video pitch for OneNote using children talking about glitter pens and toys in the OneNote space, and a very brief mention of “ink to math”, which doesn’t seem to exist in the app, but is available in OneNote in the browser. It’s a digital ink converter that recognises mathematical symbols.

Module 6: Accessibility in OneNote

This module is more on accessibility features and functions in OneNote - a button you can click that tells you “everything is OK” in your documents.

Module 7: Translation

Self-explanatory, this one: using the share button on any document within OneNote allows you to export it through Microsoft’s Translator application if you have it installed. I don’t because it isn’t available for OSX, and it doesn’t seem to be available in the browser version of OneNote, either.

The quiz

Just as challenging as the last one. Earned myself a “Certified Microsoft Innovative Educator” badge for having completed 2 hours of online Microsoft Educator CPD.

Review of progress and next steps

What I have learned in the past few hours (a lot more than the two hours these two courses suggest) is that Microsoft tools are most definitely not the way forward for my hybridised teaching in the autumn. Too much time is required in trouble-shooting problems, bugs, inadequately documented differences between platforms, and design flaws. I want a piece of chalk, not a stick of cheese and some certified training in how to use it on a blackboard.

Your mileage may vary, of course: if you are fully embedded within the Redmond way of things, then good luck to you. This particular teacher is going to be innovative in other ways, and not be burdened by having to work this difficult software. Class Notebook is not going to feature in my teaching.

I completed the rest of the OneNote Teacher Academy learning path (without taking notes, digital or otherwise) and earned my badge.

MIE certificate

My transcript is available here.

Footnotes

  1. Later, I got this to work by signing out in the OneNote app, connecting to the University VPN, signing in to the university’s EASE authorisation (in the browser), and then restarting the OneNote app, and signing in (to Office 365) in the app. That is just such a ridiculous waste of time messing around trying to figure out why it doesn’t work, only to have it suddenly behave itself at the point of launching the laptop out the window. For reasons of mental wellbeing, I am heading back to avoiding Microsoft software as far as I possibly can. Life is just too short for this nonsense. 

Microsoft Tools for Education⤴

from @ @cullaloe | Tech, tales and imagery

Having spent time thinking about the principles and approaches I will be taking with my teaching in the new semester, now I must move to thinking about the tools that I will be using.

Goodbye, Moodle

For the past 3 years, I have used a Moodle instance to support and communicate with students in preference to the University’s BlackBoard Learn. The main reason for this (there are several) is that Moodle is user-centric with features like search. Learn doesn’t have that, and is one of those systems that is designed for the designer, not the user.

Moodle has increasingly become more costly to sustain because of a series of ongoing bugs, most recently following the upgrade to 3.9, search has stopped working. Now, I’m a busy person and don’t have time to debug this right now, so I have decided that I need to use a more stable system of tools.

Community

One of the single most important aspects of learning is the community you learn with. Former students have continued to access their Moodle course for a long time after the PGDE ends. I don’t want to abandon them, or break that community, so I have set up a Slack workspace for former PGDE Physics students, including those more senior who did not have the Moodle experience. So far, they have responded well to the idea. I will be using Slack to connect to the new cohort, outside of the official channels which they cannot join until they matriculate in September. This will give them a head start on building community.

Conformity

There are a lot of people using Microsoft tools, and the university is no exception: we make increasingly consistent use of Sharepoint, Office365, OneDrive, OneNote, and the ever-present PowerPoint. Now, whilst there is a very good argument not to be suckered into the global brand domination of Microsoft, you can’t say that Bill Gates has kept all the money for himself, so there is a reason to go with the flow. The other, closer to home, reason is the perceived equity of the student cohort. Our students hate difference in experience, perceiving it to be somehow uneven or even discriminatory. It isn’t, and never has been. It’s good pedagogy to try to meet the particular needs of your learners, so teachers will always try to adapt and innovate to make things better for their own, very special, learners. That’s why I have operated a Moodle site outside of the mainstream experience for my Physics cohorts in the past 3 years.

I’m going to try to use the MS tools because the university has spent a lot of money providing them. Because it’s what the rest of the team are doing, and I might be able to offer support to my colleagues as we move forward into hybridised teaching together. Because the time cost of operating a maverick set is unnecessary, when I have the cost of conforming to bear also.

Getting ready

I need to skill up. So, I’m going to pick up an MIE, starting with the OneNote Teacher Academy course. I’m starting here because it’s where I need to begin in shifting content from the old VLE into something newer: not Learn, because it’s not fit for that particular purpose, but OneNote. Here we go.