Open Source Library Systems: Getting Started
(c) 1999 by Dan Chudnov
note: a significantly edited version of this article appeared in the August 1, 1999 issue of Library Journal. You might prefer their edited version to this version. You are free to reproduce the text of this version for any purpose and in any format, provided that you reproduce it in its entirety (including this notice) and refer to the url from which it is available: http://oss4lib.org/readings/oss4lib-getting-started.php
The biggest news in the software industry in recent months is open source. Every week in the technology news we can read about IBM or Oracle or Netscape or Corel announcing plans to release flagship products as open source or a version of these products that runs on an open source operating system such as Linux. In its defense against the Department of Justice, Microsoft has pointed to Linux and its growing market share as evidence that Microsoft cannot exert unfair monopoly power over the software industry. Dozens of new open source products along with regular news of upgrades, bug fixes, and innovative new features for these products are announced every day at web sites followed by thousands.
The vibe these related events and activities send out is one of fundamental change in the software industry, change that alters the rules of how to make software--and how to make money selling software. What is all the noise about, and what does it mean for libraries?
Open Source: What it is and Why it Works
If you've ever used the internet, you've used open source software. Many of the servers and applications running on machines throughout the wired world rely on software created using the open source process. Examples of such software are Apache, the most widely used web server in the world, and sendmail, "the backbone of the Internet's email server hardware." [TOR] Open source means several things:
- Open source software is typically created and maintained by developers crossing institutional and national boundaries, collaborating by using internet-based communications and development tools;
- Products are typically a certain kind of "free", often through a license that specifies that applications and source code (the programming instructions written to create the applications) are free to use, modify, and redistribute as long as all uses, modifications, and redistributions are similarly licensed; [GPL]
- Successful applications tend to be developed more quickly and with better responsiveness to the needs of users who can readily use and evaluate open source applications because they are free;
- Quality, not profit, drives open source developers who take personal pride in seeing their working solutions adopted;
- Intellectual property rights to open source software belong to everyone who helps build it or simply uses it, not just the vendor or institution who created or sold the software.
More succinctly, from the definition at www.opensource.org:
"Open source promotes software reliability and quality by supporting independent peer review and rapid evolution of source code. To be certified as open source, the license of a program must guarantee the right to read, redistribute, modify, and use it freely." [OSS]
Software peer review is much like the peer review process in research. Peer review bestows a degree of validity upon the quality of research. Publications with a high "trust factor" contribute ideas in published works to the knowledge base of the entire communities they serve.
It is the same for software. As described in the seminal open source work, "The Cathedral and the Bazaar" by Eric Raymond, author of the popular email program fetchmail, the debugging process can move faster when more individuals have both access to code and an environment in which constructive criticism is roundly welcomed. [ER] This leads to extremely rapid improvements in software and a growing sense of community ownership of an open source application. The feeling of community ownership strengthens over time because each new participant in the evolution of a particular application-- as a programmer, tester, or user--adds their own sense of ownership to the growing community pool because they are truly owners of the software. This community effect seems similar to the network effect seen across the internet, whereby each additional internet user adds value to all the other users (simply because each new user means there are more people with whom everyone else might communicate). For open source products which grow to be viable alternatives to closed-source vendor offerings, this growing community ownership begins to exert pressure on the vendors to join in. [NYT]
This tendency shares a striking similarity to the economic value of libraries. A library gives any individual member of the community it serves access to a far richer range of materials than what that individual might gather alone. At an extremely low marginal cost to each citizen expensive reference works, new hardcover texts, old journals, historical documents and even meeting rooms might be available through a local library. The library building, its collections, and its staff are infrastructure. This infrastructure serves as a kind of community monopoly in a local market for the provision of information. Instead of reaping monopoly profits for financial gain, however, a library returns the benefits of its monopoly to individual users. The costs of maintaining this monopoly are borne by the very community which holds the monopoly. To the extent which this model works in a given community, a library is a natural yet amenable monopolistic force. If this sounds mistaken, consider whether your community has libraries which compete or cooperate.
Library Software Today
No software is perfect. Office suites and image editors are pretty good; missile defense systems are, for all we know, appropriately effective; search engines could use improvement but usually get the job done. While there is constant innovation in library software, for many of us online catalog systems mean a clunky old text interface that often is less effective than browsing stacks. Often, this is due to the obstacles we face in managing legacy systems; new systems might be vastly improved, but we are slow to upgrade when we consider the costs of migrating data, staff retraining, systems support, and on and on. Sometimes, new versions of systems we currently use are just not good enough to warrant making a switch.
This is not surprising. The library community is largely made up of not-for-profit, publicly funded agencies which hardly command a major voice in today's high tech information industry. As such, there is not an enormous market niche for software vendors to fill our small demand for systems. Indeed the 1997 estimated library systems revenue was only $470 million, with the largest vendor earning $60 million. [BBP] Because even the most successful vendors are very small relative to the Microsofts of this world (and because libraries cannot compete against industry salary levels), there are relatively few software developers available to build library applications, and therefore a relatively small community pool of software talent.
What are we left with? Some good systems, some bad. Few systems truly serve the access needs of all of our users, failing to meet a goal--access for everyone--that most public libraries strive to achieve at more fundamental levels of service. Because libraries are community resources, we tend to be quite liberal about intellectual and physical access issues, including support of freedom of speech and ADA-related physical plant modifications. At the same time, librarians are very conservative about collections and data (remember the difficult issues when you last weeded?). Is it not odd, then, that market forces lead us to be extremely conservative about online systems software? After all, online systems are no less about access to information than having an auto-open front door or an elevator in a library building.
We read of exciting technological innovations in library-related systems. Innovations in advanced user interfaces and metadata-enabled retrieval environments and other areas have the potential to make online access more and more seamless and easy to use. Our systems, though, are too old--or not standardized enough, or too familiar to change--to take advantage of these advances. And creative ideas from exciting research seems not to make headway in real systems.
Libraries, if they indeed hold the kind of community monopoly described above, might do well to enhance their services by leveraging community-owned information systems--which open source seems to promise.
Open Source and Libraries
How could open source improve library services? First, open source systems, when licensed in the typical "general license" manner, cost nothing (or next to nothing) to use--whether they have one or one thousand users. Although the costs of implementing and supporting the systems on which software runs might not change, imagine removing the purchase price of a new search interface (or ILL tool, or circulation module, etc.) from your budget for next year. Rather than spending thousands on systems, such funds might be reallocated for training, hiring, or support needs, areas where libraries tend toward chronic shortfalls.
Second, open source product support is not locked in to a single vendor. The community of developers for a particular open source product tends to be a powerful support structure for Linux and other products because of the pride in ownership described above. Also, anyone can go into business to provide support for software for which the very source code is freely available. Thus even if a library buys an open source system from one vendor, it might choose down the road to buy technical support from another company--or to arrange for technical support from a third-party at the time of purchase. On top of this flexibility, any library with technical staff capable of understanding source code might find that its own staff might provide better internal support because the staff could have a better understanding of how the systems work.
Third, the entire library community might share the responsibility of solving information systems accessibility issues. Few systems vendors make a profit by focusing their products on serving the needs of users who cannot operate in the windows/icons/menus/pointer world. If developers building systems for the vision impaired and other user groups requiring alternative access environments were to cooperate on creating a shared base of user interfaces, these shared solutions might be freely built into systems around the world far more rapidly and successfully than ever before.
A Three-Step Process
If you are still reading, you probably suspect something here might be a good idea. You might even want to help make ideas discussed above happen. Where to begin?
Understand the Phenomenon
Axiomatic business notions have shown weaknesses throughout the information age; the utility of the internet for knowledge sharing demanded rethinking of what constitutes an information product. If nothing else, it is important for the international community of librarians to understand the open source phenomenon as part of the technology-driven shift in our understanding of the nature of information. Because the ethos and style of the open source initiative is so akin to the traditions of librarianship we hold at the core of our professionalism, we should find within open source the appropriate points of entry for the similar service and resource-sharing objectives we choose to achieve every day.
The seminal works on open source are mostly technical, but they provide an envigorating view of the current state of software engineering. All are available on the internet, and they form a core of knowledge that might one day be fundamental to our discipline. "The Cathedral and the Bazaar," by Eric Raymond [ER], is widely cited as the pivotal tome describing the technical and social processes open source entails. "The Open-Source Revolution," by Tim O'Reilly [TOR], founder of O'Reilly and Associates, Inc., a highly respected publisher of pragmatic computer-related titles, gives a broader view of the social phenomenon, in particular relating open source software development to the scientific method. Finally, www.opensource.org is a central point of focus for the Open Source Initiative. It is led in part by Mr. Raymond and appeals to both the technical and non-technical sides of the community.
To foster communication regarding open source systems in libraries, we have created a web site, www.med.yale.edu/library/oss4lib, and a listserv, firstname.lastname@example.org. They are intended as forums for announcement, discussion, and sharing of broad information; look for instructions on how to join the list along with a list of current open source projects for libraries the oss4lib site.
Use Open Source Systems Where You Are
Armed with understanding, we can find opportunities to leverage existing open source systems in our own institutions. The Linux operating system [LINUX], Apache web server [APACHE], and MySQL database [MYSQL] form a powerful, free platform for building online systems. Consider the value of these and other open source systems when making design and purchase decisions at your institution; you might find tremendous savings and increased product performance at the same time.
Beyond merely using open source products, however, we must create them. Are you already working on any new applications at your institution? Perhaps you've put a year or two into a homegrown search interface, or an online reference services tool, or a data model and retrieval code for an image archive. Is there a good reason why you wouldn't want to share that work? For those of you who realize that someone else might benefit from what you've done--and that you might benefit from the ability to share in the work of others--consider thoroughly the implications of releasing your code under an open source license. [FH] If the benefits outweigh the negatives, get started sanitizing and documenting your code as well as you can, and set it free.
Another ideal opportunity at this stage is for library and information science researchers to open their projects up for the entire community to review and develop as appropriate. Grant-funded systems builders might find an afterlife for their work by releasing their source. Faculty might design courses around building a retrieval system or improving an existing open source tool. Indeed this model is already widely used by computer science professors--at Yale, for instance, undergraduate students might work on aspects of the Linux kernel in their Operating Systems course.
Grow the Phenomenon
As the library community moves in this direction, there will be many roles for individuals in our profession to fill. Most visible is application development; there is a major need for software engineering resources to be devoted to creating community-owned library systems. This does not in any way marginalize those of us who are not programmers or database administrators. In the open source community there exists a tremendous need for exactly the skills librarians have always used in making information resources truly useful. In particular, systems testing, evaluation, and feedback to open source designers is welcome and even sought after; documentation for open source systems is always needing improvement; instructional materials for open source products are often lacking. These are all areas in which librarians excel. For the more technically minded among us, www.freshmeat.net provides constant updates and announcements of general open source projects replete with contact information for those wishing to participate. For all of us, the oss4lib listserv and website will highlight additional library-specific opportunities as they come around.
Playing a role in the larger open source community will strengthen our ability as professionals and service providers to understand how best to shape our own systems. Additionally, it might make significant inroads in demonstrating how the ethics and practice of librarianship is more vital to the movement of information than ever before. As the software industry shifts to appropriately incorporate open source models, systems in other industries might even grow to utilize products the library community creates.
An argument I have already heard against these ideas is based on experience: "We tried building our own OPAC in the eighties--it was an impossible project and we gave it up after a few years because it just cost too much." In 1999, however, we know that the internet has changed the landscape. Because it is so very easy to share ideas and software and code using the internet, software developers have already found that the old way of doing things--particularly building monolithic homegrown systems in our own institutions--makes no sense anymore. As the open source vision and culture continue to mature, librarians would be remiss not to find our profession playing a major role in that culture. For all we have done so far, our online systems are not good enough yet. We can do better.
[APACHE]Apache Server Project
[BBP] Barry, J, Bilal D, and Penniman WD. "The Competitive Struggle," Library Journal, April 1, 1998, p. 43.
[ER] Raymond, ES. "The Cathedral and the Bazaar"
[FH] Hecker, F. "Setting up Shop: The Business of Open-Source Software," Aug 3, 1998. [MS] Microsoft analysis of open source
[GPL]GNU General Public License
[MYSQL]MySQL Home Page
[NYT] Harmon, A and Markoff J. "Internal Memo Shows Microsoft Executives' Concern Over Free Software", New York Times, November 3, 1998, Sect. C, pg. 8, col. 1.
[OSS]Open Source Initiative Home