The OTW Board is pleased to announce the newly updated Archive of Our Own Roadmap.
The Roadmap is a broad outline of planned improvements and changes to the AO3. It is maintained by the Accessibility, Design, & Technology committee with input from many OTW committees including Abuse, Internationalization & Outreach, Open Doors, Support, Systems, Tag Wranglers, and Translation. While such an involved cross-committee process takes time, we are happy that it includes input from so many committees because we believe that collaboration results in a high-quality document and (hopefully!) a shared understanding of how and why we do the things we do.
Updating the Roadmap is a huge task which involves:
- Talking to everyone to learn what is needed and wanted (which can be different for everyone but thankfully there is some common ground)
- Talking about priorities (because our wish list is way larger than our ability to grant wishes)
- Working out what we need to do to get from A to C (sometimes that means people get B first on the way to C even though B was low priority)
AD&T walked the Roadmap through multiple drafts (the first rough outline was created in August 2012) and worked with multiple versions of the Board (there were five new Board members in this period) to reach a final version. The Board would like to thank everyone who worked on this task for the energy (and patience!) given to make this possible. We really appreciate all the time the staff and volunteers put into this.
A Look Back
When we started working on the Archive in 2008, we drew up a roadmap of core functionality which included all the basic posting and searching features, the tagging system, collections, comments, and a translatable interface. The original roadmap was drafted with a view to completing this groundwork within a short timeframe, and producing a releasable package which other fans could download and use. However, once we started working on the goals defined by the roadmap, the landscape changed a little. Implementing these basic, interlocking features turned out to be more complex than we originally anticipated, and as a result, the whole timeframe stretched, leaving openings where new features crept in and new urgent needs arose.
We revised our Roadmap in 2010 to give users an outline of planned features and milestones which took into account the realities of developing the live site, and the demands of our userbase. While we've been following roughly that plan, you'll see from looking at it that we didn't stick to it exactly: we have to be flexible and modify the pace of work and the order of tasks based on user demand, the needs of the site, and the skills and interests of available coders. In general, the pace of development has been slower than we anticipated, although in some cases we have been able to bring forward features which were in high demand, while keeping up with increasing performance challenges as the userbase expanded.
Some of the features we planned to add after version 0.8 have already been implemented, including a "kudos" option which lets you quickly show that you liked someone's work without leaving a full comment, improvements to editing multiple works, a download function for individual works in different formats, and better HTML styling thanks to work skins. A public Feature Requests Board was established in the form of a third-party tool (see the related Internal Tools FAQ for more information).
In contrast, several features we planned for before the 0.9 release were pushed back. For example, we've made some excellent progress on multi-language support for the Archive interface, but realized that the latest version of Ruby on Rails (the framework the AO3 is built on) offers some key features which can be used for that functionality, so we'll be upgrading the site to the new version of Rails before the new translations code is implemented. This is a big task which affects code across the site, so it needs to be carefully scheduled.
We review our Roadmap every few years to consider our needs and priorities going forward. This post aims to provide an updated timeline of our major milestones: while we'll continue to be flexible and may implement things in a different order, this roadmap gives an overview of what we've got planned. If there are features not listed here which you'd like to see, please let us know in the comments to this post! You can also vote for suggested features on our Feature Requests Board - this helps us know what people would like us to prioritize (although sometimes other constraints mean we can't implement popular requests as fast as we'd like, or can't commit to adding them to the Archive at all).
The Road so Far...
If you're interested in how the Archive has developed over the years, you can see all the work we completed up until June 2010 on the previous version of our Roadmap: everything up to Version 0.7 had been completed at the time it was published. We've listed the highlights of the work we've done since then below; for more detailed information on exactly what we did when, you can browse our extensive Release Notes, going back to the very first code deploy.
Interlude: Rails 3
In November 2010, we updated our version of Rails to Rails 3. Rails is a web framework for application development; the code of the Archive uses Rails and its various code libraries as well as other open source frameworks and tools. (For more information, see our Technology Stack page.)
This update involved changes to almost every part of the Archive code and was a major undertaking! Updating to use this version gave us access to a whole host of better features, and ensured that the codebase as a whole stayed up-to-date - important for stability and maintainability.
Version 0.8 included a huge number of changes and stretched over multiple code releases, because it covered so many complex and interrelated things. It was completed in July 2012 and included:
- Lots of performance-enhancing features, including widescale improvements in caching across the site. We introduced Squid for whole-page caching, more granular caching for frequently updated elements such as comment numbers on works, the caching of indexes for searches and sorting, and much more. These performance improvements helped us accommodate a massive expansion in the number of users in 2012.
- A major overhaul of all our CSS and frontend code, for greater maintainability and accessibility.
- The introduction of site skins to enable users to customize the appearance of the site.
- The introduction of work skins to allow users to use CSS to style their works.
- The introduction of embeds for multimedia works (fannish video and audio works like vids, AMVs, or podfic).
- The launch of email subscriptions for authors and works, and feeds for tags across the site, to make it easier for users to get updates about works they were interested in.
- The introduction of a user stats page, including many often requested features such as the ability to compare the numbers of kudos and comments on different works.
- A range of tools for our Open Doors team, enabling the import and maintenance of at-risk archives. The first archive to be imported using these tools was the Smallville Slash Archive.
- Lots of improvements to our challenge functionality, including the introduction of prompt memes!
The Road Ahead
The following is a rough outline of our plans for the next few years. We haven't included a timeframe for these milestones - not even a very rough one - because experience has shown us that a whole number of factors can influence development speed and priorities. Plans for the 1.0 release and beyond are kept very high-level right now; we will post an updated, more detailed plan as we get closer to the finish line.
The first major part of Version 0.9 was the launch of our new tag filters, along with a number of other significant changes to the underlying code of our search and browse features. The site had outgrown the old filters, which were causing performance issues, and the new code has proven more robust and more usable. It also enabled us to implement filtering for bookmarks, which was a long-needed first step on the way to making personal bookmarks a truly useful Archive feature.
Throughout this development cycle, we're also also focusing heavily on performance improvements and long-term scalability (that is, how easily the site can grow to accomodate new users). As part of this, we are making several hopefully invisible changes under the hood:
- Upgrading to the latest version of Ruby on Rails.
- Increasing the amount of caching we do throughout the site and ironing out any remaining issues related to caching.
- Adding new hardware, and optimizing our server setup.
We already added some new servers in January 2013, and you can read more about plans for expansion in our post, Going forward - our server setup and plans, part of our 2013 Spotlight on Systems.
One major code change we'll make during this release will be the complete refactoring of our work model (the large chunk of code that deals with creating, posting, and managing works). For obvious reasons, this is one of the oldest parts of the code on the site, and it needs some major revisions to make it robust, maintainable and flexible going forward. This is a very knotty piece of work (internally we refer to the work model as Mordor) and will likely take a while to get right.
The refactoring of the work model is an essential part of one of our major priorities: moving towards true support for visual art as well as fannish audio and video on the Archive. Hosting different media types (e.g. enabling image or audio file uploads to our servers) is still some way in the future, but version 0.9 will include a number of enhancements aimed at making the site more suitable for diverse types of fanworks, including better search and posting tools for non text-based media. We already allow embeds of art, fan videos, and audio works, and the first major enhancement in this release cycle will be the option to specify a media type for your fanwork when posting (instead of relying on Additional Tags alone), and to filter by media type when browsing. At the same time, we will add options for posting and searching for non-fiction fanworks (fannish meta and other non-fiction types). Our aim will be to make it as easy as possible for users to find the kinds of fannish content they are interested in, and for creators to tag and categorize their works usefully.
We will also be focusing on our collections suite, including challenges and prompt memes, to fix remaining bugs, address usability problems, and improve documentation as well as performance.
- Stability, performance, and scalability improvements
- Improvements to searching and browsing, including better filtering for media type, category, and language
- A new header and footer layout, for easier site navigation
- Improvements to tag wrangling management features, and enhancements to the tagging system to allow for improved handling of international sources and fandoms
- Refinements to the collection and challenge code, including major bug fixes for gift exchanges, prompt memes and tag sets
- Review and rewrite of existing code to prepare for art, video, and audio posting, and non-fiction fanworks
- Improvements to the handling of work relationships (remixes, translations, gifts...) across the site
- Improved and expanded tools for Open Doors to allow for easier imports of at-risk archives
Within the 0.9 release cycle, we want to take some scheduled time to review and assess our code base, documentation, resources and plans. During this time, active development will be kept to an absolute minimum and we'll avoid launching any new features. We will continue to monitor the site for any emergencies and keep sending out invites to people on the waiting list, but will focus most of our attention on laying the groundwork for good stability going forward. This is something which we always try to factor into our plans, but during very busy periods it's easy to build up technical debt and harder to make sure everything is shipshape and well-documented. This interlude will help ensure both our code and our procedures are well organized and sustainable.
In these code updates, we will build upon and enhance the multimedia support and interface translation features. Providing the Archive structure and navigation in any language, not just English, was one of our main goals when we started dreaming up plans: we will now make sure that all site elements are translatable and complete the necessary translation tools for volunteers. This is different from support for individual fanworks in languages other than English, which we hope will be vastly improved by the changes and bug fixes planned in version 0.9.
Additionally, more subscription options will be added, enabling users to keep track of exactly the kind of fanwork they want to read, look at, or listen to. Subscription management will be improved.
We will also make our admin system more robust and allow for clearly defined roles in site and user management, with improved admin features for each area (such as better tools for our Abuse staff).
In this release cycle, we will also add private messaging, one of the most-requested features.
- Translation features for an Archive in many different languages
- Private messaging features
- More subscription options
- Anonymous posting
- Improved drafts management
- Better chapter management, including the ability to indicate prologues and epilogues
- Improvements to the History feature
- Groundwork for a more modular admin system to spread out tasks (managing users, approving new public skins, posting news etc.) across distinct admin roles
- Profile enhancements, such as the option to list your presence on other services (Dreamwidth, DeviantArt, etc) and offer beta services
In our last planned release cycle before announcing 1.0, we will focus on ironing out any remaining bugs, review existing features, refactor and optimize our codebase for scalability and expandability, and make sure that, above all else, the Archive is accessible to all users. We will also review our wide range of automated tests to ensure that all our features have complementary test suites and that our testing framework adheres to current best practices before going forward.
Version 1.0 and beyond
Our goal for Release 1.0 is to have a stable package with all core features working well, and a full testing suite. This will provide a good foundation for the move to hosting fanwork files on our own servers, which will be a core goal for our post 1.0 development. It will also enable us to bundle an install package for easy installation by users without substantial systems experience. This will include the further enhancement of admin roles to allow a master admin and more site customisation settings. We also want to provide API hooks for developers to build upon our codebase for posting/bookmarking/reading/etc. tools and enhancements. A more structured outline of goals will be posted before we reach this biggest of milestones.
- A robust feature set for posting, managing, and browsing different kinds of fanworks: text (such as fic or essays), images (such as fanart, graphics, or crafts), audio (such as podfic, audio fic, fannish songs, or filk), and video (such as AMVs and vids)
- A translatable interface that makes it possible to customize the Archive for a variety of languages
- A default interface that is functional, accessible, and attractive (in that order of importance) in a wide variety of browsers, platforms and assistive tech, and easily customizable for special needs and preferences
- Public API
- Release package
How You Can Help
The Organization for Transformative Works is always looking for volunteers to donate their time and skills (or alternatively, hard-earned cash) to the development of the Archive of Our Own. You can read more about how to contribute or get in touch with the Volunteers & Recruiting committee to offer your help. Depending on open positions, we will welcome skilled programmers, database specialists, sysadmins, designers, QA personnel and technical writers, as well as interested people who are new to coding and software testing. Training and support will be provided, so anyone can help shape the future of the Archive!