Three Objections to Learning Objects and E-learning Standards
This is a draft version of a paper published in: McGreal, R. (Ed.). 2004. Online Education Using Learning Objects. London: Routledge. Pp. 59-70.
Norm Friesen (firstname.lastname@example.org)
April 7, 2003
"Learning objects," "learning object metadata," and "learning object repositories" are terms that have been central to many discussions, projects and funding priorities of both public and private educational organizations. These words have been associated with a range of benefits and technical concepts, some of which would rightfully seem strange in many educational contexts--such as the realization of systems "interoperability" and of resource "reusability," the elimination of "non-tariff barriers" to trade, and the customization of "application profiles." On the basis of the benefits that these terms might suggest, government and industry are spending substantial amounts of money, giving rise to a veritable "educational object" and "standardization" movement in educational technology.
This paper outlines a number of problems associated with this movement, all of which arise in some way from the juxtaposition of narrow technical and specialized concepts with the general and varied dimensions and contexts of learning. In doing so, it will place special emphasis on the contexts proper to public education. It raises these objections to learning objects in the interests of fostering an open discussion that would bring the concepts and thinking associated with them to more fruitful relationship with the practices of learning and education.
As I write this paper, governments around the world are spending large sums of money on initiatives that promise the development of learning objects, learning object metadata and learning object repositories to store both this data and these objects. In plainer language, learning objects can be said to refer to digital educational resources; metadata refers to their systematic description to facilitate searching and administration; and repositories represent online, searchable collections of these resources. Examples of initiatives underway include the Curriculum Online project being undertaken for schools in the UK at a cost of approximately $500 million, and the Australian Learning Federation, a project similar in emphasis with a $30 million budget. Similar projects are also being currently undertaken in Canada (e.g. eduSource, 2003; SchoolNet, 2003), the US (e.g. HEAL, 2003; iLumina, 2003) and by regional and international consortia (e.g. Universal Brokerage Platform, 2003).
Only recently has discourse in this area moved beyond broad generalization, technical elaboration, or promotional pr?cis. To the knowledge of this author (and as clamed by Banks ), there have been no in-depth studies of the pedagogical consequences of these systems and ways of thinking, and no examinations of their epistemological and ideological implications. On a more practical level, others have noted a general lack of adoption of these technologies by both practitioners and vendors (e.g. Robson, 2003; Farance, 2003).
This paper seeks to address these problems by summarizing a number of concerns that have already been raised in connection with learning objects and associated technologies, and by outlining a number of further, outstanding issues related to the vision of learning objects, e-learning standardization and the milieu from which it has arisen. It does so from the perspective of someone who has been actively involved in these activities and this environment for a number of years, and with a special emphasis on the interests and values of public education. It does so also in recognition of the fact that only through open discussion of both positive and negative aspects can the vision of sharing educational resources be made more relevant to the work of learning practitioners and to learners themselves.
Objection 1: What's a learning object, anyway?
The problems presented by learning objects and related technologies begin with the definition of the term "learning object" itself. The particular meaning or meanings associated with this term have been the subject of much debate and discussion (e.g. Wiley, 2001; Conole, 2002). Often cited in these attempts is an early definition provided by the IEEE Learning Technology Standards Committee: "Any entity, digital or non-digital, which can be used, re-used and referenced during technology-supported learning" (IEEE, 2001). Such a definition--as the IEEE itself says--implies that learning objects can include "multimedia content, instructional content, instructional software and software tools [and] in a wider sense...learning objectives, persons, organizations, or events" (IEEE, 2001). There are few things, in other words, that can not be learning objects. As David Merrill puts it, "No one seems to know what a learning object is in the first place. One of the absurd definitions I heard was, 'as small as a drop, as wide as the ocean.' In other words, if everything is a learning object, then nothing is a learning object" (Merrill, as cited in Welsch, 2002). The result of all of this, as Rehak and Mason describe, is confusion:
Different definitions abound, different uses are envisaged, and different sectors have particular reasons for pursuing their development. In this environment of uncertainty and disagreement, the various stakeholders are going off in all directions. (2003)
However, at the same time, the term "learning object" carries discernable indications of its origin and own intrinsic characteristics. The term was "first popularized by Wayne Hodgins in 1994 when he named the CedMA [sic; Computer Education Management Association] working group 'Learning Architectures, APIs and Learning Objects'" (Polsani 2002). CEdMA, in turn, describes its own purpose as the provision of a "forum" for discussion of "issues in computer training" (CEdMA, 2003). The origin of this term in the world of technical systems training is significant, as will be discussed later.
|The term "learning object" suggests neither simplicity, compatibility nor any obvious relative advantage over prevailing teaching practice.|
More important at this point is the fact that the term "object" in "learning object" has clear origins in "object oriented" programming, design, analysis and theory (Robson, 1999; Bratina, Hayes, Blumsack, 2002). This programming and design approach or "paradigm" (Alhir, 1998) has been developed and consolidated in the area of software programming and design over the last thirty years. It "started with [the programming language] SIMULA-67 around 1970," and "became all-pervasive with the advent of C++, and later Java" (FOLDOC, 2003). The form of design and analysis that goes by this name now also has considerable influence in the groups that are responsible for technical standards in e-learning. This influence is most clearly registered in their adoption of formal description techniques based on object-oriented modeling, especially UML (Unified Modeling Language). Both this object-oriented description technique, as well as object-orientation generally, are based on "such principles as abstraction, concurrency, encapsulation, hierarchy, persistence, polymorphism, and typing" (Microsoft Press, 1997).
It is at this point that this paper's first objection to learning objects comes to the fore: Namely, the fact that the term "learning object" juxtaposes two words that are in many ways incongruous and ultimately, incommensurable: The first, "object," is a thoroughly and very specific technological paradigm--as specialized terms such as "concurrency," "polymorphism" and "typing" indicate. It is part of an approach whose basic principles are so specialized as to be difficult to express in everyday language. And the second, "learning," is equally extreme in its vagueness, generality and broadly non-technical nature. In clear contrast to the dominance of the object-oriented paradigm in programming and software design, there is no consensus among educational experts as to how learning occurs or how it can best be understood. There is no "all-pervasive" approach or "paradigm" for learning or education as is claimed for programming and software design. "Pedagogy as well as instructional design," as Allert, Dhraief, and Nejdl say, "are ill-structured domains" (2002).
It is this incongruity and incommensurability separating the terms "object" and "learning" that can be seen as underlying the confusion and divergence in defining the term "learning object"--and ultimately, as contributing significantly to their slow uptake by vendors and practitioners. (It may also be a contributing factor in problems associated with learning object contextualization and the "paradox" of learning object reuse, as pointed out by Wiley and others [Wiley, 2003; Wiley, Recker, Gibbons, 2000].)
This incongruity can also be seen to repeat a negative historical pattern that has recurred in different forms with previous innovations in educational technology. In this pattern, these innovations are introduced into educational contexts and practices clearly bearing the stamp of their technical origin. Instead of being presented in terms familiar and meaningful to educators, they bear connotations that appear unclear or even negative in these practical contexts. Next in this pattern is the appearance of various forms of resistance to these innovations on the part of practitioners. Finally, this is followed by teachers and other practitioners being blamed for their resistance and inflexibility in not adopting such innovations. Speaking specifically of research into school education, Larry Cuban describes this recurring pattern as follows:
Since the mid nineteenth century the classroom has become home to a succession of technologies (e.g., textbook, chalkboard, radio, film, and television)…. Yet the teacher has been singled out as inflexibly resistant to "modern" technology, stubbornly engaging in a closed-door policy toward using new mechanical and automated instructional aids…. Seldom did investigators try to adopt a teacher's perspective or appreciate the duality of continuity and change that marked both schools and classrooms (Cuban, 1986; pp. 2, 6)
Using a term that make sense only in abstruse technical discussions, and that is opaque and confusing to practitioners does not make its potential benefits clear to teachers. Instead, it presents the potential of pitting those responsible for instruction unproductively against those advocating technological change. It is not that the innovation should not come from outside of education, or that it can only come from within. It is simply that innovations must be presented in terms that are meaningful for teaching practice.
A substantial tradition of research into the spread or diffusion of innovations among populations underscores this point. This research shows that the rate of adoption increases significantly when innovations possess some of the following characteristics: 1) simplicity, 2) compatibility with existing methods and techniques, and 3) relative advantage in comparison with these established methods and techniques (Rogers, 1962).
Innovations such as e-mail or mobile phones provide good examples of technologies that meet these requirements. Although they are new and quite different from the technologies which they supersede, their very names provide a simple and direct comparison with these established technologies. Instead of suggesting the complexity of technological abstraction, the terms "e-mail" and "mobile phone" connote the relative advantage presented by these innovations--the mobility of cellular communication, and the instantaneous nature of electronic telecommunications. "Learning objects," on the other hand, suggest neither simplicity, compatibility nor any obvious relative advantage over existing teaching practice.
In order for the positive potential of learning objects to be realized, they need to be labelled, described, investigated and understood in ways that make the simplicity, compatibility and advantages claimed for them readily apparent to teachers, trainers and other practitioners.
Objection 2: Where is the Learning in E-Learning Standards?
National and international committees, consortia and other organizations have been busy developing standards and specifications for e-learning technologies at least since the late 1990's. They have been doing so with the understanding that the benefits of this standardization work will be manifold and variegated:
Not only would the development and use of international standards [in e-learning] produce a direct cost savings, but the information technology systems could be used in a wider range of applications, and used more efficiently. Better, more efficient and interoperable systems, content, and components will produce better learning, education, and training – which has a positive effect upon all societies. (ISO, 2002)
Organizations actively developing these standards and specifications include the IMS Global E-Learning Consortium, the IEEE Learning Technologies Standards Committee, and the ISO Subcommittee on "Information Technology for Learning Education and Training." The development of technical standards in e-learning can be understood as a part of the maturation of this emergent field, sector or industry. Before and especially since the popular emergence of the Internet and the World Wide Web, digital technologies have been used widely in education--both in distance and classroom education as well as off-line and online training settings. However, these technologies have typically been applied in ad hoc and divergent forms: Innumerable courses, course components and systems for managing and delivering these courses have been developed independently of one another, often at great expense. Moreover, this content and these management systems are often created in a manner that makes it very difficult if not impossible to support their interchange or their successful interoperation. Standards in e-learning seek to address these shortcomings by ensuring the interoperability, portability and reusability of this content and of these systems.
Such rationales imply that the standards and specifications are not allied with any one approach or paradigm associated with learning. Just as the protocols and standards ensuring interoperation on the Web (e.g. http or HTML) can support many kinds of documents and types of information, e-learning standards and specifications are expected to be able to support multiple forms and practices of learning. Accordingly, they are frequently described as "pedagogically neutral" or "pedagogically agnostic" (e.g. Conole, 2002, p. 7; IMS, 2003a).
One prominent specification effort that emphasizes this type of neutrality is the "SCORM" initiative (Shareable Courseware Object Reference Model): It describes itself as providing "a pedagogically neutral means for designers and implementers of instruction to aggregate learning resources for the purpose of delivering a desired learning experience" (2002 p. 2-3). It seeks to accomplish this not be generating standards of its own, but by simplifying, combining and bringing into interrelation a number of existing specifications and standards. SCORM is being developed by the Advanced Distributed Learning initiative (ADL), an effort sponsored by the White House Office of Science and Technology Policy and the US Department of Defence. The SCORM framework or reference model is intended to make a key contribution to the ADL's mission, which is "to provide high quality instruction and decision aiding anytime, anywhere and tailored to each learner’s needs" (ADL, 2001a, p. 1-11). The ADL further describes its raison d'être as follows:
The ADL initiative assumes that dependable and efficient instruction and decision aiding will adapt itself to the unique needs, abilities, background, interests and cognitive style of each learner. It will tailor the content, pace, detail, difficulty, etc. of its presentations as needed by specific individuals at specific times (ADL, 2001a, p. 1-12).
However, despite (or perhaps because of) these ambitions goals, a number of those implementing or closely associated with SCORM have been expressing misgivings regarding its relevance and value. In contradistinction to SCORM's own claims to "pedagogical neutrality," some of these experts have been asserting that SCORM is limited in its pedagogical scope, neutrality or relevance. Dan Rehak, one of the "chief architects" behind SCORM, has stated that this framework, has "a limited pedagogical model unsuited for some environments" (as cited in Kraan & Wilson, 2002). "SCORM," Rehak says, "is essentially about a single-learner [whose learning is] self-paced and self-directed. This makes it inappropriate for use in [higher education] and K-12" (Kraan & Wilson, 2002). Thor Anderson, a key contributor to many of the specifications incorporated into SCORM, states that the SCORM framework has so far done a good job of meeting the needs of "vendors with the content they have right now." But Anderson emphasizes that in order for this content to be made "instructionally useful and pedagogically relevant, we have to modify [the] path" that is currently being followed (as cited in Welsch, 2002).
It is the contention of this paper that these issues arise not from the particularities of SCORM's or ADL's approach to standards and specifications, but from its implied understanding of pedagogy: namely, from its simultaneous claims to pedagogical relevance and pedagogical neutrality.
|specifications and applications that are truly pedagogically neutral cannot also be pedagogically relevant|
The very meaning of word "neutrality"--the state of "not assisting, or actively taking the side of" (OED, 1987)--implies a state or position that is antithetical or perhaps even anathema to pedagogy and teaching--the act of appropriately "guid[ing] studies" or "show[ing] by way of information or instruction" (OED, 1987; Merriam-Webster, 2003). The active engagement implied in pedagogy and teaching, in other words, does not admit of the non-involvement and impartiality that is implied in the words "neutral" and "neutrality." Also, understood more abstractly as a domain of knowledge and research, pedagogy as a whole is not something that can simply be understood as neutral in its relation to technology or technical specification. As a hetergenous and "unstructured field" (Allert, Dhraief, & Nejdl, 2001), the mere term "pedagogy" or "pedagogies" includes areas as diverse as critical pedagogy, performance support, special needs education, home schooling and so on. Each of these approaches to or contexts for pedagogy, moreover, presents various predispositions and factors that would shape its particular relationship to technology and e-learning standardization. Simply put, specifications and applications that are truly pedagogically neutral cannot also be pedagogically relevant.
Objection 3: Education in a Militarized Zone?
Learning objects and e-learning standardization bear the imprint of the ideology and culture of the American military-industrial complex--of ways of thinking that are related either marginally or antithetically to the interests and values of education generally and public education in particular. In The Classroom Arsenal, Douglas Noble identifies the characteristics of what he calls the "military worldview" that have shaped the approach of the US Department of Defense to training technology. Noble begins his study by emphasizing the preponderance of military educational research over its public counterpart in the United States:
within government agencies, the military spends seven dollars for every civilian dollar spent on educational technology research. Each year, for example, the military spends as much on educational technology research and development as the Department…of Education has spent in a quarter century." (1991, p. 2)
Using technical systems and weaponry of ever-increasing complexity, the US Department of Defence attempts to address its ever-growing training needs by employing the same approaches to education as are used in the development and deployment of weapons and command-and-control systems. Not surprisingly, characteristics of the military worldview in general reappear in its approaches to education in particular. Noble describes these characteristics and his own treatment of them as follows:
The three principal…characteristics of the 'worldview'… of US military research and development to be addressed here are 1) technological innovation 2) command and control, and 3) systems thinking. Subsidiary topics that are also discussed include the military research emphasis on speed, change, and efficiency; uniformity and standardization; task specificity; and simulation. (1991, p. 5)
The characteristics of this worldview that are of special significance in this paper are its emphasis on "systems thinking" and, of course, the prominence of "uniformity and standardization."
In military contexts, "systems thinking" is often realized in the form of cybernetic, "man-machine" weapons systems (the gender exclusivity of this last military phrase is deliberately retained). Indeed, cybernetics--"the comparative study of automatic control systems" in biological and artificial entities (Merriam-Webster, 2003)--had its origin in military efforts to "theorize humans as component parts of weapons systems" (Edwards, as cited in Noble, 1991). This approach is still very important today. In a recent paper of the same title, Schneider and Lt Col Braskin argue more broadly for an understanding of "learning" in general "as a weapon system" --an approach they explain as follows:
In a nutshell, it means treating the development of [Air Force] personnel as a weapon system. A weapon system is created using a systems engineering approach and best commercial practices. The Department of Defense defines systems engineering as the design and management of a total system, which includes hardware and software, as well as other system life-cycle elements. [Similarly, learning] is based on operational needs and requirements. It contains instructional systems, exercises, experiences, and processes that require development, integration, and management to efficiently and effectively develop the necessary operational proficiencies in [Air Force] personnel. ....Humans, by their very nature are multi-mission capable. This capability needs to be harnessed and nurtured (2002).
The end result of this approach is to understand training and the technologies that support it as a means of "engineering" and maximizing the performance of the human components of a larger system. The performance of these human components can then be fine-tuned and optimized in a manner similar to the way their mechanical and electronic counterparts are maintained and refined.
"...such training is considered within military training research to be a personnel corollary to advanced technological development; its principle objective is to fit human components--gun crews, computer operators, pilots, electronics technicians, equipment repairers, communications specialists, clerical and managerial support, etc.--into complex man-machine weapons systems and information systems by the most efficient means possible. Therefore training is conceived in the military as a species of human engineering." (Noble, 1991, p. 48)
The understandings from which such an approach is derived are clearly emblematized in the diagram provided here. It is originally entitled "The ADL Model," and closely resembles diagrams found in SCORM documentation.
|From: Slosser, S. (2001) "ADL and the Sharable Content Object Reference Model." MERLOT 2001.|
Through its depiction of varied content (screw-driver, soldier, teacher) being subjected to the machine processing of a "learning management system," and subsequently being communicated to the relevant organ of the "student," "professional" or "warfighter," this diagram depicts without apparent exaggeration a vision of e-learning and e-learning specification that US military sees as attractive. This vision is one where components--human organs, machine components, devices and objects--work together as a larger system, with each part engineered to achieve optimal performance.
In this light, it is perhaps not surprising that many of the key participants in the e-learning standardization movement--Rehak and Hodgins, for example--are engineers themselves, or have close ties with engineering organizations (Hodgins, for example, is a "Strategic Futurist" at Autodesk, makers of AutoCAD software for mechanical engineers).
|The obvious fact is that the goals of public education are radically different than those of the American military.|
Such an emphasis on the understandings and methods of engineering lead directly, as Noble himself explains, to a corresponding emphasis on technical standards and specifications as solutions to pedagogical problems. If these problems can be understood simply as deficiencies in engineering and design, then their solution lies in the development of rigorous technical standards, and the instantiation of these standards in strictly conforming systems and implementations:
A system fulfills some purpose or intended use, and in order to do so a system must meet certain standards or criteria of operational effectiveness.... This, in turn, requires precise specificity of component design; the Department of Defense is notorious in industrial production for the rigor and idiosyncratic character of its "military specifications" or "milspecs." (Noble, 1991, p. 34)
Public education, despite its radically different goals and levels of funding, can be seen as having embarked on a larger, idiosyncratic "edspecs" enterprise. It is participating actively in specifications development, and seeking to build content and infrastructure according to these specifications. However, given the nature of its funding and goals, the ability of the public education sector to support an "edspecs" project--modeled on military and engineering precedents--seems doubtful. In addition to being smaller by several orders of magnitude, funding for educational technology research in public education is typically short-term and project-based (e.g. Roschelle, 1996). The development of specifications, standards and corresponding implementations, on the other hand, is a costly, long-term undertaking. A typical standard will have a development lifecycle of five or more years (Farance, 1999); and multiple standards, following different and sometimes ill-defined development timelines (e.g. Friesen, 2003; Kraan, 2003), are required for the successful interoperation of objects, repositories and other systems. Add to this the costs of conformance testing and certification that standardization often entails (e.g. IMS 2003b), and the price tags of conformant systems and content may well exceed many public institutions' and private individuals' budgets. It is little wonder that some have been calling for spending on a military scale --"a 'Manhattan project' approach, with sustained major funding over a decade or longer"-- for the realization of an "advanced," "collaborative" learning object infrastructure (York et al, 2002; see also Duval & Hodgins, 2002). As far as public education is concerned, standards are currently an unproven commodity, involving investments and investment risk that is not frequently acknowledged.
E-learning specifications--as well as the military and engineering connotations they carry with them--thus present an important challenge to public and other forms of education: Namely, to carefully and accurately identify the acceptable limits of their relevance and costs. As Noble urges, such an examination of the relevance of technical specifications in public education--like any examination of approaches engendered in a different context--must be undertaken with great care:
…while appearing to address… problems in public education, [military educational technology] research actually participates in an entirely different enterprise, one with marginal and antithetical import for education. This is the design and engineering of man-machine systems. (1991, p. 146)
Looking at the man-machine system schematized in the diagram above, it is easy to see that vision of e-learning specifications and systems for military applications is rather far afield from the practices, contexts and values that dominate public (and other forms of) education. The obvious fact is that the goals of public education, however they might be construed, are radically different than those of the American military.
For the full potential of e-learning standardization and infrastructure efforts to be realized, it is important that these efforts place significantly greater attention on existing educational practice, on issues of innovation adoption, and on the heterogeneity of educational activities and contexts in general. To properly deal with this divergence and complexity--and with issues also now emerging from training and other communities--it is necessary to look beyond systems engineering techniques and standardization processes. These techniques and processes may work well for more exclusively technical applications, but they are proving inadequate for dealing with the ambiguities implied in education and even in the deceptively simple term "learning." They also bring with them a culture and set of connotations that are (at the very least) not entirely helpful in public education. Perhaps most importantly for e-learning content and standardization, it is important to recognize that objects and infrastructures for learning cannot simultaneously be both pedagogically neutral and pedagogically valuable. Developers and designers will have to recognize and choose relevant (and probably differing) pedagogical positions, or risk pedagogical irrelevance.
ADL. (2003). Sharable Content Object Reference Model Version 1.2: Conformance Requirements. http://www.adlnet.org/ADLDOCS/Documents/SCORM_1.2_ConformanceReq.pdf
ADL .(2001b). Sharable Content Object Reference Model Version 1.2: The SCORM Overview. http://www.adlnet.org/ADLDOCS/Documents/SCORM_1.2_Overview.pdf
ADL. (2001b). Sharable Content Object Reference Model Version 1.2: The SCORM Content Aggregation Model. http://www.adlnet.org/ADLDOCS/Documents/SCORM_1.2_CAM.pdf
Alhir, S.S. (1998). UML in a Nutshell, Sebastopol, CA: O'Reilly & Associates.
Allert, H. Dhraief, H. Nedjl, W. (2002) Meta-Level Category 'Role' in Metadata Standards for Learning: Instructional Roles and Instructional Qualities of Learning Objects. COSIGN 2002 The 2nd International Conference on Computational Semiotics for Games and New Media. http://www.kinonet.com/conferences/cosign2002/pdfs/COSIGN2002.pdf
Banks, B. (July 2001). "Learning Theory and Learning Objects." http://www.fdlearning.com/fdlearning/html/company/features/l- theory-l-objects.pdf
Baskin, R. R. & Schneider, D. L. (2002). "Learning as a Weapon System." I/ITSEC 2002 Conference Proceedings. http://www.iitsec.org/authordir/2002_PMR_Learning%20as%20a%20Weapon%20System.PDF
Canada's Schoolnet. (2003). http://www.schoolnet.ca/home/e/
CEdMA. (2003). Computer Education Management Association. http://www.cedma.org/guestabout_cedma.html
Conole, G. Systematising learning and research information Journal of Interactive Media in Education, 2002 (7) http://www-jime.open.ac.uk/2002/7
Cuban, L. (1986). Teachers and machines the classroom use of technology since 1920. New York: Teachers College Press.
Curriculum Online. (2003). http://www.dfes.gov.uk/curriculumonline
Duval, E. & Hodgins, W. A LOM Research Agenda. http://www.cs.kuleuven.ac.be/%7Eerikd/LOM/ResearchAgenda/ResearchAgenda.html#VanDam2002
eduSource. (2003). eduSource Canada. http://www.edusource.ca
Farance, F. (2003). "IEEE LOM Standard Not Yet Ready For 'Prime Time.'" Learning Technology (Publication of the IEEE Computer Society Learning Technology Task Force [LTTF]). 5 (1). January 2003. http://lttf.ieee.org/learn_tech/issues/january2003/index.html#8
Farance, F. (1999). Standards and Specification Development Organizations. http://ltsc.ieee.org/ppt/standards-process.ppt
FOLDOC. (2002). Free online dictionary of computing. http://wombat.doc.ic.ac.uk/foldoc/
Friesen, N. (2003). Standardization Progress: A Report from IEEE LTSC Meetings. http://phenom.educ.ualberta.ca/n/report_22.03.03.html
Heal. (2003). Health Education Assets Library; National Multimedia Repository. http://www.healcentral.org/index.htm
IEEE. (2001). IEEE Learning Technology Standards Committee (LTSC) IEEE P1484.12 Learning Object Metadata Working Group; WG12 Home page. http://ltsc.ieee.org/wg12/s_p.html
iLumina (2003) iLumina: Educational Resources for Science and Mathematics. http://turing.csc.uncwil.edu/ilumina/homePage.xml
IMS. (2003a). IMS Learning Design Information Model: Version 1.0 Final Specification. http://www.imsproject.org/learningdesign/ldv1p0/imsld_infov1p0.html
IMS. (2003b). Facility for Interoperability Testing FITness Testing Directory. http://www.imsglobal.org/icpfit/index.cfm
ISO bulletin, June 2002. Information Technology: Learning by IT. http://jtc1sc36.org/doc/36N0264.pdf
ISO, 2002. "Information Technology: Learning by IT." ISO Bulletin. June 2002. http://jtc1sc36.org/doc/36N0264.pdf
ISO. (2003). ISO 9000: The Basics. http://www.iso.ch/iso/en/iso9000-14000/tour/beginnin.html
Kraan, W. (2003). IEEE to lift SCORM, IMS Content Packaging to standard status, clarifies LOM future. http://www.cetis.ac.uk/content/20030203181028
Kraan, W. & S. Wilson (2002). Dan Rehak: "SCORM is not for everyone." http://www.cetis.ac.uk/content/20021002000737
Maddocks, P. (2002). "Case Study: Cisco Systems Ventures into the Land of Reusability." Learning Circuits -- ASTD's Online Magazine All About E-Learning. March, 2002. http://www.learningcircuits.org/2002/mar2002/maddocks.html
McGraw, K. L. (2001). "E-Learning Strategy Equals Infrastructure." Learning Circuits -- ASTD's Online Magazine All About E-Learning. June 2001. http://www.learningcircuits.com/2001/jun2001/mcgraw.html
Merriam-Webster Online (2002). http://www.m-w.com/
Microsoft Press. (1997). Microsoft Press Computer Dictionary (3rd ed.). Redmond, WA: Microsoft Press.
Noble, D. D. (1991). The Classroom Arsenal: Military Research, Information Technology and Public Education. New York: The Falmer Press.
OED. (1989). Oxford English Dictionary. Oxford: Oxford University Press.
Polsani, P. R. (2003). Use and Abuse of Reusable Learning Objects. Journal of Digital Information. (3) 4. http://jodi.ecs.soton.ac.uk/Articles/v03/i04/Polsani/
Rehak, D. & Mason, R. (2003). "Keeping the Learning in Learning Objects." Reusing Online Resources: A Sustainable Approach to eLearning. A. Littlejohn (ed.) London: Kogan Page.
Robson, R. (1999). Object-oriented Instructional Design and Web-based Authoring. http://robby.orst.edu/papers/objectoriented.html
Robson, R. (2003). "Information products, learning products, context and the role of standards in increasing the value of content." Learning Objects and Metadata. McGreal, R. (ed.) London: Kogan Page.
Rogers, E.M. (1969). Diffusion of Innovations. The Free Press, New York, New York.
Roschelle, J., and Kaput, J. (1996). Educational Software Architecture and Systemic Impace: the Promise of Component Software." Journal of Educational Computing Research, 14 (3), 217-228, 1996.
Slosser, S. (2001) "ADL and the Sharable Content Object Reference Model." MERLOT 2001.
The Learning Federation. (2003). http://www.thelearningfederation.edu.au
Bratina, T. A., Hayes, D. and Blumsack, S.L. (2002) Preparing Teachers to Use Learning Objects The Technology Source (November/December 2002).
Universal. (2003). The Universal Brokerage Platform for Learning Resources. http://www.educanext.org/UNIVERSAL
Welsch E. (2002). SCORM: Clarity or Calamity? Online Learning Magazine 2002-07-01 http://www.onlinelearningmag.com/training/search/search_display.jsp?vnu_content_id=1526769
Wiley, D. (2001). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. The Instructional Use of Learning Objects. Wiley, D. (ed.) http://www.reusability.org/read/chapters/wiley.doc
Wiley, D. (2003). Learning Objects: Difficulties and Opportunities. http://wiley.ed.usu.edu/docs/lo_do.pdf
Wiley, D. Recker, M. M., & Gibbons, A. (2000). The reusability paradox. http://rclt.usu.edu/whitepapers/paradox.html
York, B., van Dam, A., Ullman, J., Soloway, E., Pollack J., Kay, A., & Kalil, T. A teacher for every learner. June 2002. http://www.cra.org/Activities/grand.challenges/slides/education.pdf.