Ian Robertson is a Senior Lecturer at Royal Melbourne Institute of Technology (RMIT) University, Victoria, Australia, with 20 years experience in vocational education. His main areas of interest relate to teaching practice and technology.

This peer reviewed article describes a study that sought to explore the scope of teachers’ use of Information and Communication Technology (ICT) in the Vocational and Technical Education (VTE) setting in Australia. An instrument was developed which identified a detailed set of functionalities in relation to ICT use supporting learning. The instrument also included a means to record frequency of use. The instrument was implemented among a representative sample of VTE teachers and the responses analysed to discover patterns and themes in usage. The findings revealed widely varying levels of ICT usage between those surveyed and varying levels of usage of the various functionalities. The article suggests a number of factors as determinants of ICT usage among VTE teachers: newness, complexity, compatibility and locus of control. It argues that consideration of these criteria would influence the use of online learning technologies when applied to the design of professional development interventions and activities for VTE teachers.

Introduced by Professor Ron Oliver of Edith Cowan University.

Click to download the print version Surveying Online Technology: A matter of design by Ian Robertson.doc (171kb) or Surveying technology:a matter of design pdf(115kb).

Surveying Online Technology: A matter of design

Abstract

This article argues that previous attempts to survey the use of technology in vocational education and training in Australia have been based on broad descriptors such as online learning and e-learning. These lack definitional clarity rendering the results open to wide interpretation. The article reports on the development, administration and analysis of a survey to determine the frequency of use of 21 functionalities by 116 vocational education and training teachers. As might be expected, some functionalities are used more frequently than others. The article explores the patterns of use frequency and proposes that the criteria of newness, complexity, compatibility and locus of control are influential in determining how frequently the respondents use e-learning functionalities.

Introduction

Established in 2000, the Australian Flexible Learning Framework’s 2000-2004 strategic plan stated that by 2004, Australia would be recognised as a global leader in applying new technologies to vocational education and training products and services (Australian National Training Authority 2000). The most recent business plan for the Australian Flexible Learning Framework describes its purpose as ‘…[t]o increase the sustainable uptake of quality e-learning in vocational education and training’ (Department of Education 2005:3). In order to achieve this aim, the use of technology will need to be systemically embedded into the teaching practices of vocational practitioners. Assessment of this outcome requires data related to the frequency of use of technology for teaching and learning purposes.

Early surveys of the uptake of technology in vocational education and training suggested that adoption rates were less than 2.5% (Hill et al. 2003; National Centre for Vocational Education Research 2002). General critiques of these studies identified definitional confusion, a lack of reliable data and methodological differences as causing difficulties (Brennan, McFadden & Law 2001; Cashion & Palmieri 2000; Kilpatrick & Bound 2003).

Terms related to online technology in education are often used interchangeably (Booker 2000). Even single terms such as online learning and online delivery are subject to multiple definitions and interpretations (Brennan et al. 2001; Hill et al. 2003). Recently the term e-learning has emerged. Here again there is a lack of definitional clarity. Some limit the scope of e-learning to the use of networked technologies and the Internet (Garrison & Anderson 2003; Gillani 2003). Others define e-learning as a broader notion to include both networked and non-networked technologies (Australian National Training Authority 2003; Bowles 2004). To add further confusion the term blended learning (Bersin 2004) has become more common in reference to the use of online technology in teaching.

The most recent national survey of technology uptake in Australia, the 2005 E-learning Benchmarking Project used a broad definition of e-learning as ‘…access to, downloading and use of web, CD-ROM or computer-based learning resources in the classroom, workplace or home’ (I & J Management Services 2005:5). It is reported that 60% of 478 vocational teachers across Australia have used e-learning in the last 12 months and surveyed the use of a range of functionalities as summarised in Table 1.

Whilst the 2005 E-learning Benchmarking Project report provides some useful data, the project survey’s use of a broad-based definition of e-learning renders the results open to wide interpretation. The survey has the following limitations.

1. Frequency of use is not evaluated. The collection of data on teacher’s use of e-learning is defined as a single use in the last 12 months. If one is attempting to identify the systemic embedding of technology in teaching such a figure is of little use. For example, it could mean that the 60% of respondents have used a single aspect of e-learning once in 12 months.
2. Some functionalities surveyed require significant interpretation on the part of the respondent. For example, information is collected on the ‘Use of multimedia interactive learning resources in the classroom’ and ‘Online access to and participation on course activities’. These are broad concepts that lack specificity.
3. Only a yes/no option is provided in respect to use. It is not possible for the respondent to indicate that they are unsure.
4. The survey collects demographic data related to gender, age, State/Territory and main field of vocational and technical education (VTE) teaching. Given the diverse nature of the VTE sector and the range of employment modes, the addition of further demographic descriptors such as years in teaching, provider type, primary mode of teaching and level of technical skill would allow more detailed analysis according to demographic segment.

This article reports on the use of a survey to determine the frequency of use of 21 functionalities in the Australian Flexible Learning Framework’s 2005 Networks Community Forum (hereafter referred to as the Community Forum). The Community Forum aims to establish ‘…sustainable professional learning practices within an environment of online networking, knowledge sharing and knowledge management’ (Australian National Training Authority 2004:22). It is made up of over 500 participants mainly working within the Australian VTE system with others from universities, schools and industry. There is a small group of international educationalists.

The following section describes the design, administration and analysis of the survey. Factors that influence the levels of use of the various functionalities are explored and four criteria are proposed.

Survey design and administration

An online survey was made available to vocational practitioners registered with the Community Forum in November 2005. Members were forwarded an email inviting them to participate in the survey and Internet links were placed on the Community Forum website. The survey is divided into three sections. Part one offers the opportunity for the respondent to identify themselves by name and email address. Of the 136 people who responded 85 (62.5%) provided their name and 81 (59.6%) provided their email address.

Respondents were asked to indicate if they were employed in a teaching role which was defined in the instructions to the survey as follows:

• interaction between teacher/trainer and learner(s), may be face-to-face, at a distance or in combination
• the use of networked technology for learner support, teaching or assessment
• networked technology implies technology that is linked to an intranet or the Internet i.e. NOT stand alone computers.

In order to gain access to the second part of the survey, respondents were required to identify themselves as ‘teachers’. According to the three descriptors above, a positive response indicates that the respondent considered they were engaged in interaction with learner(s), and used intranet or Internet based networked technology to support learning, teaching or assessment. Of the 136 people who entered the survey 116 (85.3%) indicated that they had a teaching role and were given access to the second part of the survey. Those who indicated that they were not involved in a teaching role were thanked for their assistance and refused entry to the remainder of the survey.

Part two of the survey collected demographic data relating to: time fraction involved in teaching; place and location of employment; primary mode of teaching; gender and age; years of teaching experience; level of technical skill; and primary vocational discipline area. These data provide the opportunity to analyse the frequency of use of online functionalities according to demographic categories. The data are reported here but no attempt is made to realise the potential of segmented analysis.

Part three of the survey collected data related to the frequency of use of 21 online functionalities. These were derived from personal experience, identifying those functionalities that are known to be used by vocational practitioners or functionalities that are promoted for use through the Australian Flexible Learning Framework. Whilst there is a lack of definitional clarity in terms used to describe the use of online technology, the functional attributes for teaching are more readily identified. These include content presentation, communication, information seeking and construction of communities of learners (Gillani, 2003). Automated assessment is another functionality of technology that should be added. The functionalities surveyed are consistent with these categories and detailed in Table 2 (to follow). Respondents were able to indicate that they used the functionality daily, weekly, monthly, less than monthly, never or were not sure. The design of this part of the survey was based on the principles that the survey should list functionalities using language that would be familiar to teachers rather than technology experts, and frequency criteria that would resonate with the respondent group.

Findings

Demographic data

Using available 2002 data from all Australian States and Territories a national survey of the vocational and technical education workforce found that data was inconsistent between jurisdictions (NCVER 2004). However, using the available data it was broadly observed that Technical and Further Education (TAFE) teachers are composed of a roughly equal split according to gender. There are more part-time and non-permanent staff than full-time or permanent staff. Few staff are less than 30 years old, and between two thirds and thee quarters of TAFE teachers are older than 40 years with a relatively even split between 40-49 year olds and 50+ year olds. More than 40% have been employed in TAFE for more than 15 years (NCVER 2004).

Between 108 and 116 respondents to the current survey provided details for each of the demographic criteria surveyed. This indicates that some respondents did not complete all questions. The data show that approximately 70% of respondents are employed in a TAFE organisation (public provider), approximately 16% in a private Registered Training Organisation (RTO) provider, the remainder (approximately 14%) in Adult and Community Education (ACE), a university or secondary school. The respondent group is characterised as follows:

• gender balance is approximately 60% male, 40% female
• approximately 40% of respondents are in the 40-49 year old age range and 40% are 50 years or older, the remainder (approximately 20%) are less than 40 years old
• approximately 45% teach full-time, 23% one day per week and the remaining 32% between two and four days per week
• almost 60% have been teaching for at least 10 years, 28% for 5-9 years and the remainder (approximately 12%) for less than 5 years.

Whilst only 70% of the respondents to the survey are TAFE teachers, at a broad level it would seem reasonable to propose that this respondent group is not largely dissimilar in demographic profile of the general TAFE teacher population reported by NCVER (2004) in relation to gender, employment status and length of employment.

The current survey also finds that approximately 63% describe their primary mode of teaching as face-to-face (possibly with some non-face-to-face), 23% as primarily non-face-to-face, and 14% as a balance of face-to-face and non-face-to-face teaching. The use of this data provides the opportunity to segment the survey findings according to primary mode of delivery but that is beyond the scope of this article.

The survey reveals that only 3.5% of the respondents indicated that they are beginners with the use of technology, the remainder were approximately equally split in describing themselves as intermediate, experienced and advanced in their level of technical expertise. General opinion would suggest that more than 3.5% of vocational teachers would rate their level of technical skill with online technology as a beginner. Given the common interest of participants in the Community Forum in flexible learning and the use of online technology, it is possible that the group may have a higher level of technical skill than the general population. As a consequence the results of this survey may not be entirely representative of the general vocational and technical education teaching population. The survey does however demonstrate the potential of a survey approach that deconstructs technology based practices into functionalities.

Use of e-learning functionalities

Between 99 and 103 respondents indicated the frequency with which they used each of the functionalities surveyed. This means that not all respondents to the demographic survey also responded to the survey of functionalities. Figure 1 graphically depicts the frequency of use of each functionality in percentage terms. Table 2 summarises the frequency of use of each of the 21 functionalities from most to least frequency of use on a daily plus weekly basis.

These findings could be used to provide a benchmark against which future surveys are conducted. The data could also contribute to a consideration of return on investment for institutions that fund the development, implementation and maintenance of such functionalities. For example, the knowledge that Flexible Learning Toolbox products are used by approximately 19% of vocational practitioners on a daily or weekly basis may be of interest to government authorities that have funded the development of these products.

Table 2: Frequency of use of functionalities (daily plus weekly) (click top view full size image).

The next section of this article explores the frequency of use and the nature of the functionalities surveyed to reveal criteria that influence usage patterns.

Factors influencing the frequency of use of technology

Rogers’ (1995) in Diffusion of Innovations describes a general theory of innovation adoption derived from a range of traditions in sociology, health, communications, and economics. Rogers describes five innovation characteristics that influence the likelihood of adoption: relative advantage; compatibility; trialability; observability; and complexity.

1. Relative advantage relates to the perceived benefit that will accrue as a result of adoption in comparison to existing practices. The greater the perceived relative advantage, the greater the likelihood of adoption.
2. Compatibility refers to the level of congruence with the individual’s existing values and beliefs, previously introduced ideas and compatibility with needs. The greater the compatibility the greater the likelihood of adoption.
3. Trialability refers to the degree to which experimentation is possible. In cases where it is possible to trial the innovation adoption is more likely.
4. Observability refers to the degree to which the result of adoption can be observed. The more observable the results the more likely is adoption.
5. Complexity, relates to the perceived complexity of the functionality. Innovations that are perceived as less complex are more likely to be adopted and functionalities perceived as more complex are less likely it is to be adopted.

In a more specific consideration of the adoption of information technology Davis (1989) found that ‘perceived usefulness’ and ‘perceived ease of use’ were significantly correlated with both self-reported current and predicted future use. He proposed that the perceived ease of use may be a causal antecedent to perceived usefulness (Davis 1989).

Common to the views of both Rogers (1995) and Davis (1989) is the notion that the adoption of technology is influenced by the individual’s perception of the technology. That is, the technology is more likely to be adopted if it is perceived in a positive light. Whilst logical reasoning may influence the person’s perception, it may not be adequate to convince the individual that there is a perceived benefit.

Davis’ (1989) idea of perceived usefulness is consistent with Rogers’ (1995) criteria of relative advantage and compatibility. That is, a technology is more likely to be perceived as useful if the practitioner perceives a relative advantage over existing practice and if the technology is compatible with that practice. Davis’ (1989) idea of ease of use is consistent with Rogers’ (1995) idea of complexity. That is, a technology is more likely to be perceived as easy to use if it is simple to use rather than complex.

An analysis of the characteristics of e-learning functionalities that are used more or less frequently in the current study suggests that there are four criteria that influence adoption: newness; complexity; compatibility; and, locus of control. The first two of these criteria relate to the nature of the technology, the second two relate to teacher’s practice. My interpretations are based on personal knowledge of vocational teachers’ practice and premised on the model of largely face-to-face teaching in a TAFE college. Given that approximately 63% of respondents reported that their primary mode of teaching was face-to-face, and 14% a balance of face-to-face and non-face-to-face practice, this would seem to be a reasonable model of practice to adopt for the respondent group.

Analysis of data shows that the eight most frequently used functionalities are used at least weekly by more than one-third of respondents and at least monthly by more than one half of respondents. These eight functionalities relate to individual and group email (teacher to learners), Internet searches for learners, Internet/intranet distribution of a range of file types and electronic submission of assessment in the form of files. Analysis shows the following in respect to these eight most frequently used functionalities.

1. Newness: All have been available for some time and cannot be considered to be recent additions to vocational teaching.
2. Complexity: All are likely to be able to be used by vocational teachers with a moderate level of technical skill at their desktop and without the assistance of intermediaries.
3. Compatibility: All are likely to be compatible with, or at least not incompatible with the face-to-face teaching practices of a typical TAFE teacher.
4. Locus of control: None require the teacher to cede control over the means of communication, sequence, pace or assessment criteria to another party.

These findings cannot be universally asserted across the remaining thirteen functionalities.

Newness and Frequency of Use

Newness is related to the passage of time from when the functionality has been generally available and promoted for use in education rather than the time from development. Newness may be associated with the use of an existing technology in a new context: for example, the use of email for personal communication as compared with the use of email to support teaching practice.

In a temporal sense, some of the 21 e-learning functionalities have been generally available to support teaching and learning for some years, others have been introduced more recently. As a general trend, functionalities that have been available for some time (e.g. email, internet searches, distribution of files) are used more frequently by more vocational practitioners than those that are more recently introduced (e.g. blogs, virtual classroom software, e-journal, e-portfolio). It is reasonable to propose that the more recently that a functionality has been generally available and promoted for use in education, the more likely that the frequency of use will be at a low rate. With the passage of time, the frequency of use is likely to increase. However, there is variation in the frequency of use across functionalities that have been available for some time. That is, increases in frequency of use over time are not universal or equal in all cases.

For example, discussion forum, bulletin boards, Internet/intranet for computer based assessment, group email between learners, Flexible Learning Toolbox products and text chat have been available and promoted for use in education for some time but less than one-third of respondents report the use of these functionalities on a daily or weekly basis. This finding begs the question: what factors influence the frequency of use of online functionalities with the passage of time? I propose that the increasing use is associated with the perceived complexity, compatibility and locus of control.

Complexity

Complexity is related to the difficulty that a vocational teacher is likely to experience in using the functionality. This criterion is consistent with the notions of complexity described by Rogers (1995) and ease of use as described by Davis (1989). The less complex the technology the more likely it is to be used frequently by vocational practitioners.

The eight most frequently used functionalities surveyed require the skills of email communication, Internet searching, use of the Internet/intranet for the distribution and receipt of a range of file types. In a general sense, it is likely that a vocational teacher who is able to independently conduct tasks such as downloading and installing software, managing files and creating basic level files would be able to operate the most frequently used functionalities at their desktop without the assistance of other specialists such as computer programmers and graphic designers.

Whilst the use of the Internet/intranet for the distribution of web-based (html) learning resources may be interpreted as requiring html programming skills, the surveyed functionality refers specifically to distribution rather than development. In future surveys, the differentiation between development and distribution would be worthy of more obvious distinction. However, for the purposes of the current article I have interpreted the findings associated with this functionality in a literal sense.

Computer marked assessment and Flexible Learning Toolbox products have been available for some years for teaching in vocational education, yet approximately one-quarter or less of respondents use these functionalities on a daily or weekly basis. They clearly fall within the lower 50% of functionalities in terms of frequency of use. I propose that the use of these products requires skills beyond those held by the average vocational teacher and this, at least in part, accounts for their lack of use by many teachers.

The use of the criteria of compatibility is consistent with that described by Rogers (1995) who defines compatibility in terms of the congruence of an innovation with the individual’s values and beliefs, existing practices and perceived needs. The more compatible the functionality the more likely it is to be used frequently. The issue of change and teachers’ values and beliefs is complex and has been discussed elsewhere (Errington 2001, 2004; Pajares 1992). It is beyond the scope of the current article.

Compatibility can be interpreted in two ways. Firstly, functionalities are more likely to be adopted if they are congruent with existing values and beliefs, practices and perceived needs. That is, they are perceived to have a positive benefit. This notion is similar to the criteria of relative advantage proposed by Rogers (1995) and usefulness proposed by David (1989). The second interpretation is that functionalities are more likely to be used if they do not have a negative impact on, or require significant changes to, existing values and beliefs, practices or perceived needs. That is, there is no perceived negative impact.

The eight most frequently used functionalities surveyed require the skills of email communication, Internet searching, use of the Internet/intranet for the distribution and receipt of a range of file types. If we use the model of a vocational teacher whose practice is primarily face-to-face in a traditional TAFE college then, I contend that the eight most frequently used functionalities are, at least, not inconsistent with conventional face-to-face teaching practice. That is, even if they are not perceived to have a positive influence on teaching and learning interaction, they do not require any significant change in practice.

Beyond the eight most frequently used functionalities, a number have the potential to require a change in practice. For example, discussion forum, text chat, blog, virtual classroom software, audio and video conference would require teachers’ to operate outside of normally timetabled sessions. Some require the application of new skills and practices. This observation brings me to a fourth criterion which is not explicitly identified by Rogers (1995) or David (1989), that is, locus of control.

Locus of Control

Locus of control is related to who controls the pedagogic interactions enabled by the online functionality. Bernstein (1996) refers to the concept of framing to describe the locus of control of selection of communication, sequence, pace and assessment criteria. My proposition is that functionalities that leave control of decisions related to pedagogy with the teacher are more likely to be used frequently.

The eight most frequently used functionalities relate to individual and group teacher to learner emails, Internet searches for information, Internet/intranet for the distribution of files and electronic submission of assessment in the form of files. None of these require a teacher to cede control of communication, sequence, pace or assessment criteria to either the learners or another party. Alternatively, whilst computer marked assessment and Flexible Learning Toolbox products are not new, they require the teacher to cede control over the design of much of their program design to others.

Conclusion

This article reports the development, administration and analysis of a survey to reveal details related to the frequency with which particular functionalities of online technology are used by a group of teachers in vocational and technical education. The findings reveal differing frequencies of use of functionalities amongst the respondent group. This data has the potential to contribute to an analysis of return on investment.

Analysis of the patterns of use and the characteristics of the functionalities suggests that there are four main criteria that promote and limit the frequency of use. These are newness, complexity, compatibility and locus of control. A consideration of these criteria might positively influence the frequency with which online functionalities are used if applied to the design of interventions and activities aimed at increasing the use of technology based functionalities.

Acknowledgements

I wish to thank the Australian Flexible Learning Framework 2005 E-learning Network Community Forum for allowing me to collect the data used in this article. In particular, I am grateful to Cathy Baxter and Jyothi Jayaram for enabling the collection of the data and to the members of the Community who responded. Whilst I’m personally responsible for the published article, I am indebted to Professor Ron Oliver for his suggestions and constructive criticism which considerably improved the final product.

References

Australian National Training Authority, 2000, Flexible learning for the information economy A framework for national collaboration in vocational education and training 2000-2004, Australian National Training Authority, Brisbane.

Australian National Training Authority, 2003, Implementation plan 2003: Australian Flexible Learning Framework for the National Vocational Education and Training System 2000-2004, Australian National Training Authority, Brisbane.

Australian National Training Authority, 2004, 2005 Australian Flexible Learning Framework for the national vocational education and training system. 2005 Framework business plan, Australian National Training Authority, Brisbane.

Bernstein, B. 1996, Pedagogy, symbolic control and identity.Theory, research, critique, Taylor and Francis, London.

Bersin, J. 2004, The blended learning book. Best practices, proven methodologies, and lessons learned, Pfeiffer, San Francisco.

Booker, D. 2000, Getting to grips with online delivery, National Centre for Vocational Education Research, Leabrook.

Bowles, M. S. 2004 Relearning to e-learn. Strategies for electronic learning and knowledge, Melbourne University Press, Carlton.

Brennan, R. McFadden, M. & Law, E. 2001, Review of research. All that glitters is not gold: online delivery of education and training. National Centre for Vocational Education Research, Leabrook.

Cashion, J. & Palmieri, P. 2000, ‘Evaluation of quality of learning on line in the VET sector’. Paper presented at Open Learning 2000, Brisbane, Queensland.

Davis, F. 1989, ‘Perceived usefulness, perceived ease of use, and user acceptance of information technology’, MIS quarterley, 13, pp. 319-340.

Department of Education, Science and Training, 2005, 2006 Framework Business Plan. Australian Flexible Learning Framework for the national vocational and technical education system, Department of Education, Science and Training, Canberra.

Errington, E. 2001, ‘The influence of teacher beliefs on flexible learning innovation in traditional university settings, in Innovation in open and distance learning’ in Successful development of online and web-based learning, eds F. Lockwood & A. Gooley, Kogan Page Limited, London, pp. 27-37.

Errington, E. 2004, ‘The impact of teacher beliefs on flexible learning innovation: Some practices and possibilities for academic developers’, Innovations in education and teaching international, vol.41, no. 1, pp. 39-47.

Garrison, D. R. & Anderson, T. 2003 E-learning in the 21st century. A framework for research and practice, Routledge Falmer, London.

Gillani, B. 2003, Learning theories and the design of e-learning environments, University Press of America Inc., Lanham, Maryland.

Hill, R. Malone, P. Markham, S. Sharma, R. Sheard, J. & Young, G. 2003, Researching the size and scope of online usage in the vocational education and training sector, National Centre for Vocational Education Research, Leabrook.

I & J Management Services, 2005, 2005 E-learning benchmarking project, Final report, Department of Education, Science and Training, Canberra.

Guthrie, H. 2003, Online learning. Research findings, National Centre for Vocational Education Research Ltd., Leabrook.

Kilpatrick, S. & Bound, H. 2003, ‘Online delivery and learning in regional Australia. Benefits and barriers’, in Online learning. Research findings, ed H. Guthrie, National Centre for Vocational Education Research Ltd., Leabrook, pp.185-203.

Lockwood, F. & Gooley, A.. (eds) 2001, Innovation in open and distance learning. Successful development of online and web-based learning, Kogan Page Limited, London, pp. 27-37.

National Centre for Vocational Education Research, 2002, Flexibility through online learning, National Centre for Vocational Education Research, Leabrook.

National Centre for Vocational Education Research, 2004, Profiling the national vocational education and training workforce, National Centre for Vocational Education Research, Leabrook.

Pajares, F. M. 1992, ‘Teachers’ beliefs and educational research: Cleaning up a messy construct’, Review of educational research, vol. 62, no. 3, pp. 307-332.

Robertson, I. 2006a, ‘Deconstruction, confusion and frequency: Surveying technology use by vocational teachers’. Paper presented at the 9th Annual AVETRA Conference: Global VET: Challenges at the Global, National and Local Levels, University of Wollongong.

Rogers, E. 1995 Diffusion of innovations, 4th edn., The Free Press, New York.

Copyright © 2006 Robertson I.The author assigns to The Knowledge Tree e-journal and educational non-profit institutions a non-exclusive license to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author also grants to The Knowledge Tree e-journal a non-exclusive license to publish this document in electronic or print form within The Knowledge Tree e-journal publications and/or the World Wide Web. Any other usage is prohibited without the express permission of the author.