AO3 News

Post Header

Banner by Diane with the outlines of a man and woman speaking with word bubbles, one of which has the OTW logo and the other which says 'OTW Announcement'

There have been a lot of changes in the OTW in the past few days. The current Board of Directors has resigned, effective December 15. We know there has been some confusion, and we wanted to share with you the situation as it currently stands:

  • The OTW and its projects, including the Archive of Our Own, Fanlore, and Transformative Works and Cultures, are operating normally. Our volunteers are still carrying out their work and will continue to do so throughout this process. Rest assured that everyone's first priority is to keep the projects and the organization running smoothly.
  • We, Matty Bowers and Atiya Hakeem, new Board members elected earlier this month, will take office on December 1st. We should have access to all the tools and information available well before the 15th.
  • We have a long list of things we know need to be worked on, compiled from internal and external feedback, and our own plans for the Org. We will rely on this to prioritize and start taking action.
  • We are committed to publishing a budget for 2016 as soon as possible. An essential part of this process will be to make a draft and then seek OTW teams' feedback on the chosen items. This may not end up happening until early 2016, depending on the status of the current draft. We'll know more come December 1st, when we receive access to the draft.
  • We hear everyone's questions about the next election cycle and what happens to the vacant seats on the Board. We will do our best to have information about that out as soon as we can; we can't officially make decisions or announcements until December.

Please remember that this is a work in progress! There is a lot of information to be gathered, and a lot of work ahead. There has been a great deal of internal and external support for this fresh start and we are grateful to everyone who has reached out and offered to help. We’re currently working to organize our information and resources so we can move ahead as soon as possible.

Thanks for your support, and for the vote of confidence.

—Matty and Atiya (Lady Oscar)
Board members-elect


Post Header

Banner by Diane of a 3 line checkbox with the choices 'OTW', 'Elections News' and a checkmark next to 'Make your voice heard'

On 14 August 2015, the Board voted to change the size of the Board from 9 Directors to 7. Before this vote, the Board informed Elections that only 2 seats would be available to be filled this year. This is complicit with Article V, #3 of the by-laws, which states that at least 2 Directors must be elected yearly, and if the Board has 7 or more Directors, at least one-third of the Directors must be elected yearly.

On 16 September, the Elections Chair pointed out that, since it is possible to elect more than the minimum requirement in a given year, we had an opportunity to revitalize a shrinking Board due to the unexpected candidate turnout this year. (There was initially a total of 8 candidates.) The Board overruled this, with Andrea Horbinski replying as follows: "The Board has voted to set the size of the board at seven directors. Since we elected three people last year, we can elect two people each this year and next year and easily be on a regular schedule--the same one we were on before the size of the Board was increased to nine."

Later, the Board removed Nikisha Sanders from the ballot. Again, the Elections Chair protested and was overruled by the Board. Subsequently, Dan Lamson withdrew his candidacy.

6 candidates ran in the final election. They were voted on and ranked by members in this order:

  1. Matty Bowers
  2. Atiya Hakeem
  3. Alex Tischer
  4. Katarina Harju
  5. Aline Carrão
  6. Andrea Horbinski

The 5th place candidate, Aline, received 2.89 times more votes than the 6th place candidate, Andrea.

Today, the Board held an Open Board meeting in the publicly accessible Public Discussion channel. Despite not warning the volunteers, members, or Elections Committee that this vote was expected to take place, the Board voted in favor of Andrea Horbinski returning to the Board to finish out the term ending 31 December 2016, which was vacated at the end of 2014 by Anna Genoese. Andrea Horbinski, Margaret J. MacRae, Cat Meier, and Jessica Steiner voted in favor, Eylul Dogruel abstained, and Soledad Griffin was absent. You can read a transcript of this vote and the resulting reaction at this link.

In 2010, a similar situation occurred when a Board member retired immediately after that year's election. At that time, the next-place candidate, Hele Braunstein, was brought in to fill the seat. Precedent, common sense, and ethics dictate that the membership's choices in an election be considered and respected. The Elections Committee supports the appointment of Alex Tischer to the Board to fill Anna's seat. If Alex does not wish to take on the position, we support the appointment of Katarina Harju or Aline Carrão, in that order.

ETA November 23, 2015: The 2015 Board has resigned, effective December 15. They will be replaced by Atiya Hakeem and Matty Bowers, who will join the Board no later than December 1.

ETA2 November 23, 2015: Due to these unusual events, Elections Staff released the ranking information about this year's candidates with permission from 5 of the 6 candidates. However, as our goal is to promote an equal and harmonious Board of Directors, we do not intend to reveal this information in future elections unless a situation like the one we witnessed last weekend happens again.


Post Header

Banner by Diane with the outlines of a man and woman speaking with word bubbles, one of which has the OTW logo and the other which says 'OTW Announcement'

The OTW Board of Directors voted at its regularly scheduled meeting on 22 November to appoint Andrea Horbinski to serve the remainder of the term vacated in 2014 by Anna Genoese, ending 31 December 2016. Filling board vacancies by appointment is a normal part of board work provided for in Article V §4 of the OTW Bylaws, and the Board has done so at multiple points in the past.

After discussion with the rest of the Board, Andrea Horbinski has decided to decline the appointment to the OTW Board for 2016. She has tendered her resignation from the Board effective 15 December 2015. Soledad Griffin, Jessica Steiner, Eylul Dogruel, Cat Meier, and M.J. MacRae are also resigning from the Board effective on that date. Those who currently serve as members of OTW committees will remain with the organization in their staff roles but not their Board roles.

The 2015 Board wishes the 2016 Board and the volunteers, staff, and membership of the OTW all the best for the organization’s continued success.


Post Header

2015-11-19 17:45:32 -0500

The big news this time around is comment moderation. Many thanks to the users who helped us trial this feature, and to the Abuse team for their input! In addition, we have done some work on our automated testing infrastructure and made some corrections and improvements to our documentation.

A big round of applause and thanks to the Systems team for their hard work debugging and resolving the hardware-related downtime earlier this month!


  • Coders: Ariana, Elz, Naomi (shalott), Nerine, Sarken
  • Code reviewers: Ariana, Elz, Enigel, james, Naomi, Scott, Sarken
  • Testers: Lady Oscar, mumble, Sarken, madelyn, torigates, truffulafruits


  • [AO3-3543] - Comment moderation is here! Please read the details below for updates to our previous post, Comment moderation is coming, including some additions we made after testing and user input.
  • [AO3-4364] - A handful of the tests in our automated test suite fail intermittently due to timing issues. We have disabled caching on these tests to reduce the frequency of failures.
  • [AO3-4320] - Added a link to the more detailed Tags FAQ to the tags help files.
  • [AO3-4141] - Updated the Edit Pseud form to link to the help file with the correct set of available HTML tags.
  • [AO3-4034] - In the Diversity Statement, we were "the archive" instead of "the Archive" when referring specifically to the AO3. Now we have a capital A! \o/
  • [AO3-4373] - A new database column (to track if a work has comment moderation enabled or not) temporarily broke search indexing, whoops! Fixed.

More Detail on Comment Moderation

Turning Comment Moderation On or Off

Comment moderation can be turned on and off at the per-work level. You can set it for a new work or edit an existing work to turn it on or off in the privacy settings.

Privacy options when posting or editing a work: only show to registered users, disable anonymous commenting, enable comment moderation

You can also turn it on or off for multiple works at once using the "Edit Works" tool from your dashboard or works page.

Privacy options when editing multiple works at once: anonymous commenting, comment moderation

Approving Comments

When you have comment moderation turned on, you will have to approve all the comments before they are posted publicly. You can review comments on a work by checking its "Unreviewed Comments" page:

Row of buttons at the top of a work, including the link to access all unreviewed comments on the work

Page showing all unreviewed comments on a work, to be approved, deleted, or marked as commercial spam

You will also be able to approve comments from all your works from your inbox (assuming that you have JavaScript enabled and have not turned off your inbox.)

Unreviewed comment on the user's inbox page

You will not be able to reply to a comment until it is approved.

You cannot "un-approve" a comment, but as before, you can still delete it.

Posting Comments

When you leave a comment on a work with moderated comments, there will be a note about the moderation before and after you post.

If you are a logged-in user, you will be able to edit or delete your comment after you post it, but your comment will not be linked from the work. If you want to be able to find it again, you can either bookmark your comment or turn on the preference to receive email copies of your own comments.

If you edit your comment after it is approved, it will go back into moderation (unless the edit is very small, like fixing a typo).

If you reply to someone else's comment, they will be notified of your reply when your reply is approved.

If You Need To Report a Comment

If you receive a comment that you want to report to the Abuse team, you can use the abuse report form and include a link to your list of unreviewed comments (which can be accessed by Abuse personnel), or to the work or comment in question. Since comments can be edited if they were left by a logged-in user, we recommend taking a screenshot (which you can send as an email attachment once the Abuse team responds to your report) or holding on to the comment notification you received.

Known Issues

See our Known Issues page for current issues.


Post Header

Banner by Erin of a spotlight on an OTW logo with the words 'Spotlight on Legal Issues'

The Trans Pacific Partnership is a treaty among 12 nations: Australia, Brunei Darussalam, Canada, Chile, Japan, Malaysia, Mexico, New Zealand, Peru, Singapore, the United States, and Vietnam. Many have criticized the secretive process by which it was negotiated over the last 5 and a half years, but on October 4, 2015, negotiations ended and all of the member states agreed in principle to a treaty whose text was disclosed on November 5, 2015 by the U.S. Trade Representative.

This treaty hasn’t been signed into law yet by anyone, and it may never be. But its language and concepts may become the law in at least some of the 12 member countries - as well as other nations that sign onto it in the future - so it’s good for fans to understand the impact its intellectual property provisions would — and wouldn’t — have on fan activities. We wanted to provide this information now, as countries are considering adopting the TPP, to give fans an opportunity to review and comment on their governments’ adoption of it, if they want to.

In nearly every situation, the treaty requires member nations to strengthen protections for rights holders as a minimum floor for intellectual property rights; countries are allowed to adopt stronger protections than the treaty includes. Member nations are often allowed, or even encouraged, to make exceptions that would protect fans (and other users or follow-on creators), but they’re never required to. As a result, the treaty’s protections for rights holders are much stronger than its protections for fans. The treaty has some significant problems, discussed below, and doesn’t provide the safeguards for fans that we had hoped for. Overall, the treaty exports U.S. intellectual property laws to the other 11 countries, bringing the other countries’ intellectual property laws closer to their American counterparts.

Here are a few areas where legal changes might have an impact on fan activities in nations that sign on to the treaty. In addition to general provisions, we have noted a few particular areas of interest for fans in the US, Canada, Chile, and Japan. We will continue to watch for issues specific to other member nations.

1. Fair Use and Fair Dealing

The treaty encourages member nations to have “balance” in copyright law, which would include fair use and fair dealing exceptions to copyright infringement, but it doesn’t require them to do anything in particular to protect users or follow-on creators. For countries that already have fair use or fair dealing laws, it doesn't change anything, so it won't diminish fair use in the United States or fair dealing and the "YouTube Exception" in Canada. In general, it contains a lot of language that gestures toward a need for balance, but while pro-protection provisions are requirements, most of the provisions that support rights for users are merely suggestions. For example, the treaty provides only that parties will “endeavor to achieve appropriate balance in its copyright and related rights system,” but it gives little guidance about what balance countries might deem “appropriate.”

The treaty does allow member nations to place limitations or exceptions on copyright that give “due consideration to legitimate purposes such as, but not limited to: criticism; comment; news reporting; teaching, scholarship, research, and other similar purposes; and facilitating access to published works for persons who are blind, visually impaired, or otherwise print disabled.” These are all interests that align with existing fair use and fair dealing laws, and with many fan activities. But the treaty doesn’t require member nations to have fair use or fair dealing laws, and doesn’t even say anything about how hard members have to try to move toward copyright balance. So this isn’t necessarily bad for fans, but it isn’t necessarily an improvement or expansion of fan rights either.

2. Vidding and Technological Protection Measures

The treaty requires member nations to provide “anti-circumvention” laws that punish people for breaking “technological protection measures” (that is, encryption or DRM) on copyrighted works or making tools for doing so. It requires that the punishment for violating these laws include both civil and criminal penalties for willful violation. It also forces member countries to make circumvention illegal even if the circumvention doesn't lead to infringement. This is very similar to the anti-circumvention provisions that exist in the U.S. under the Digital Millennium Copyright Act—and it’s bad for fans, for exactly the same reasons the US provisions are bad. For example, vidders need to rely on breaking the DRM encryption on DVDs and online video sources in order to make high-quality vids.

In the U.S., the OTW has fought hard and won an exemption allowing vidders to break DVD, Blu-Ray, and online video encryption. That exemption is not at risk from this treaty, which says that member nations "may" create exceptions where there is an "actual or likely adverse impact of those measures on those non-infringing uses"—like the vidding exemption. But the treaty doesn’t require countries to pass those exceptions or create a framework creating and continuing those exceptions. Thus, member nations must import encryption rules into their copyright codes, but also can choose not to make exceptions at all, or can choose the inefficient and burdensome system we have in the U.S. The treaty also requires that when nations consider making exceptions, they also consider whether rights holders have already taken measures to allow non-infringing uses to be made. This means that rights holders might try to rely on “screen capture” and other similar technologies that they claim—falsely—allow vidders to make vids without decrypting originals.

3. The Public Domain

The public domain is the universe of works that aren’t protected by intellectual property law, either because protection has expired or because the law never protected them in the first place. The public domain is important for fans because it represents works for which fans don’t need to rely on exceptions like fair use or fair dealing to make follow-on works. The treaty states that the parties “recognize the importance of a rich and accessible public domain,” and “acknowledge the importance of informational materials…that assist in the identification of subject matter that has fallen into the public domain,” but it contains no affirmative duty on members to identify, preserve, or promote a robust public domain. This is disappointing—but although it does not require any nation to make things better for the public domain, it also doesn’t require any nation to make things worse, with the exception of copyright duration, which we describe next.

4. Copyright Duration

The treaty would require members to lengthen copyright protection so that works are protected for 70 years after the death of the author or, for works created by anonymous, pseudonymous, or corporate entities, 70 years after publication. This represents a 20-year extension from previous treaties, but it’s still shorter than U.S. copyright, which lasts for 70 years after the death of the author but lasts 95 or 120 years in the case of works created by anonymous, pseudonymous, or corporate entities. Ultimately, this 20-year extension may not make a practical difference for that many works—it’s an awfully long time either way—but it does mean that it will take longer for works to enter the public domain, which is bad for fans. The good news is that the treaty won't require member nations to yank material back into copyright protection if copyright protection has already expired.

5. Notice-and-Takedown and Fan Privacy

The treaty requires member nations to have a “safe harbor” for Internet Service Providers (“ISPs”) that implement a notice-and-takedown regime that looks a lot like the one created by the U.S. Digital Millennium Copyright Act (“DMCA”). It also requires member nations to provide legal procedures, like those in the DMCA, that allow rights holders to get information about the identities of alleged infringers so they can enforce copyright directly. There are some carve-outs for countries like Canada, which has a “notice-and-notice” system, and Chile, where ISPs are not required to take down content without a judicial order. These carve-outs are limited to those countries, though; everyone else is bound to a U.S.-style system.

Although the treaty contains some safeguards for fans, like penalties for rights holders who issue wrongful takedown notices, it remains unbalanced. It permits, but does not require, member nations to implement a “counter-notice” procedure like the one in the DMCA that allows users to have their material restored in the case of an improper takedown. Likewise, it permits, but does not require, a Japanese-style system of verification of takedown notices by an independent body.

Privacy issues also intersect with the treaty's provision on internet domain names, which requires countries to provide "online public access to a reliable and accurate database of contact information" of domain name registrants. While this provision would undermine privacy for domain name registrants, there would likely be ways to work around it. First, it would only apply to registrants in countries that sign on to the TPP, and the European Union, which has strict privacy laws, could try to block access to or distribution of this information for EU domain name registrants who are EU residents. Also, this provision might not prevent registrants from relying solutions that use "middlemen" (like GoDaddy and Register-dot-com), who could store registrants' information in their own databases, identify themselves as Domain Name Contacts, and forward communications to the actual registrants without making the registrants' identities or contact information publicly accessible to the whole internet.

6. Criminal Penalties

Finally, the treaty provides for criminal penalties for copyright infringers. This isn’t entirely new: many countries, including the U.S., provide criminal penalties for certain kinds of copyright infringement, some quite harsh. In the U.S., such penalties have mostly been reserved for large-scale piracy operations. But the treaty requires much more sweeping criminal penalties, requiring member nations to provide for criminal penalties for any act of willful copyright infringement “on a commercial scale,” even if not done for financial gain, if the infringement has a “substantial prejudicial impact on the interests of the copyright” owner in the marketplace. In fact, it goes even farther: member nations must allow “competent authorities” to initiate legal action for criminal penalties even without the need for a formal complaint by a private party or right holder. The treaty seems, however, to allow member nations to limit “competent authorities’” power to enforce copyright to situations where there is an “impact on the right holder’s ability to exploit the work in the market.”

Because copyright owners don’t object to most fan activities, the idea that authorities other than the copyright owner could enforce copyright has been a major concern for many fans during the treaty negotiation process. It was the subject of very vocal opposition by Japanese fans in particular, who were concerned that cosplay, doujinshi, and other fan activities would be subject to the nation’s already-harsh criminal copyright penalties, even though Japanese copyright owners have long allowed large-scale sales of fanworks, and even stores that primarily sell fanworks. The treaty provision isn’t as bad as some had feared, because it only applies to infringements that harm the copyright holder’s ability to exploit its work in the market. On October 5, the Japanese government released a summary of the TPP that confirmed exceptions for copyright infringement on an (unspecified) non-commercial scale. In Japan, this is being interpreted by some as a sign that lobbying by fannish organizations and legal scholars was successful and that the Japanese government wants to protect fanworks, presumably because it recognizes their social and (especially) economic value as a bedrock of Japanese manga culture. However, it's still unknown how exactly these provisions will be reflected in Japanese, and other, laws. Fears also remain that certain aspects of Japanese fan culture that are more easily interpreted as “commercial”, like fanworks shops or individual fans who sell a lot of copies of their works, will still get in trouble. So there are still a lot of interesting developments to come for Japan. And for all the member nations--not just Japan--it’s still not clear why the treaty should ever allow for enforcement of copyright by non-copyright-holders, much less require it in any circumstance.

In the end, the treaty’s criminal penalties should not be able to impact fanworks in a gift economy, or even commissioned work, because they only apply to willful infringement, and they only apply to infringement “on a commercial scale” that has a “substantial prejudicial impact” on the interests of the rights holder. For most fanworks, especially the noncommercial transformative fanworks that the OTW is focused on, none of these three things is true, much less all three. And in countries with fair use and fair dealing laws, most noncommercial fanworks works are not infringing in the first place, so criminal penalties don’t make a difference. But the criminal penalty provisions are particularly harsh for fans in countries where there aren’t fair use or fair dealing to rely on.

If you have other questions, OTW Legal is always happy to answer questions about the TPP or other issues of fandom law! Feel free to e-mail them at


Post Header

2015-11-08 12:14:49 -0500

Banner by caitie of a newspaper with the names and logos of the OTW and its projects on the pages.


On October 27, 2015, the U.S. Copyright Office released its Rulemaking regarding exemptions to section 1201 of the Digital Millennium Copyright Act. Last year, OTW's Legal Committee asked the Copyright Office to renew the exemption permitting fan video makers to circumvent the copy protection on DVDs and digital transmissions, and to expand it to include Blu-Ray discs. And on October 27, we won! The new version of the exemption, which goes into effect immediately, permits people to circumvent copy protection to obtain short portions of films, television shows, and other videos for the purpose of making noncommercial videos that criticize or comment on the original work. The exemption explicitly permits the use of "screen capture" technology and also permits circumvention of copy protection on DVDs, digital transmissions, and Blu-Ray discs if necessary to produce sufficiently high-quality content.

We’ve been working on this particular renewal and expansion effort since September 2014, and because the process requires renewals every three years, we expect we will be back fighting the same fight in another two years. In the meantime, we send out our particular thanks to the Electronic Frontier Foundation, who joined us in this effort, and to the many vidders who responded to our calls for stories and explanations of how this exemption matters to what they do.

The Legal Committee also joined allies Public Knowledge in filing a comment with the Intellectual Property Enforcement Coordinator of the U.S. Office of Management and Budget in response to that agency’s request for comments on its plan for intellectual property enforcement. The comments made specific recommendations and emphasized the importance of considering the interests of the public at large, including users who can be harmed by overly aggressive enforcement of intellectual property laws.


Our Development & Membership Committee wrapped up a successful donation drive earlier this month, raising US$171,363! We are continuing to send out donation premiums. We encourage anyone who is missing their premium to get in touch with us using the Development & Membership contact form, and we'll figure out a solution to get your premium to you. Many things were learned from this drive, and we look forward to putting them to us during the next one. The Translation Committee also successfully collaborated with Development & Membership for the drive, producing over 150 documents in 17 languages!


This month, the Accessibility, Design & Technology Committee deployed several small releases addressing miscellaneous issues, including some adjustments to our infrastructure. We were also excited to offer a preview of comment moderation, but have yet to be able to push the button due to investigating and trying to fix recent site slowness and downtimes.

The Abuse Committee received about 450 tickets in October. Most were addressed within a few days, though we did have some complicated ones come in. Please be patient if you don't receive a reply immediately — the more complex the report, the more time we'll need to spend investigating!

The Tag Wrangling Committee revised the guidelines for canonical character tags that represent groups of people (to clarify that they should not be either metatagged or subtagged to individual characters), which should help both to reduce the load on servers by getting rid of unhelpful tag trees and to help users more easily find exactly the works they're looking for.


The OTW's Board of Directors met for their Annual Meeting from October 2–4. During this meeting, the 3-year OTW Roadmap produced by the Strategic Planning Committee was approved. The meeting minutes for all three days can be found on the Board Meeting Minutes page of our website.

The Elections Committee is in the final weeks of this year's OTW Board Elections. We'd like to thank the candidates for running this year. They worked hard to answer member questions in their Q&A Responses and participate in five Candidate Chats. Chat transcripts are available on our website. Voter instructions were also sent to current members. For more 2015 Board Election updates, you can visit the Elections Committee at our Twitter, @otwelections.

The annual election is a team effort — and not just the Elections team! Many thanks go out to Development & Membership, Translation, Communications, and more. Without these and other committees helping behind the scenes, this election would not be possible. Thank you all!


The Volunteers & Recruiting Committee helped Tag Wrangling, Communications, and Internationalization & Outreach recruit for new volunteers and staffers this month. Thank you to everyone who applied, and everyone who helped spread the word! We also sent a Still Willing to Serve survey out to all current volunteers, and we greatly appreciate the responses of our continuing volunteers, as well as the service of our departing volunteers. Last but not least, we have also started inducting volunteers from our September recruitment round: a warm welcome to all new volunteers, and three cheers for current volunteers taking on new roles!

New Committee Staff: Sofia (Translation), Oriontsuki (Translation), SunnyB (Translation), Cat Goodfellow (Communications), James Baxter (Web Strategy, Design, & Development), Janet Vandenabeele (Communications), qwu (Abuse), Oniondip (Abuse), Nerine Luna Cyran (Abuse), Sammie Jarrett (Abuse), Ilona van der Vegt (Abuse), shiloh (Abuse), Miranda (Abuse), fredoxley (Abuse), 6 other Abuse, 2 Communications, 1 AO3 Documentation, and 1 Systems
New AO3 Documentation Volunteers: 1
New Tag Wrangler Volunteers: 1
New Translator Volunteers: J4Y and 3 others

Departing Committee Staff: squidgie (Wiki), Janna N. (Support), Noah Feeman (Open Doors), Jintian (Communications), Julia Allis (Communications), Ellorgast (Communications), Marta-Bee (Communications), 2 other Communications, 1 other Support, 1 Legal, 2 Elections, 1 Quality Assurance & Testing, and 1 Tag Wrangling
Departing AO3 Documentation Volunteers: briar_pipe
Departing Tag Wrangler Volunteers: Hana Lee, Asha, Swimmies, kiii17, Rana Jones, Elizabeth Moonstone, ryasdfghjkl, and 36 others
Departing Translator Volunteers: Erilia Wu and 8 others

For more information about the purview of our committees, please see the committee listing on our website.


Post Header

2015-11-04 17:53:27 -0500

At approximately 23:00 UTC on October 24, the Archive of Our Own began experiencing significant slowdowns. We suspected these slowdowns were being caused by the database, but four days of sleuthing led by volunteer SysAdmin james_ revealed a different culprit: the server hosting all our tag feeds, Archive skins, and work downloads. While a permanent solution is still to come, we were able to put a temporary fix in place and restore the Archive to full service at 21:00 UTC on October 29.

Read on for the full details of the investigation and our plans to avoid a repeat in the future.

Incident Summary

On October 24, we started to see very strange load graphs on our firewalls, and reports started coming in via Twitter that a significant number of users were getting 503 errors. There had been sporadic reports of issues earlier in the week as well, but we attributed these to the fact that one of our two front-end web servers had a hardware issue and had to be returned to its supplier for repair. (The server was returned to us and put back into business today, November 4.)

Over the next few days, we logged tickets with our MySQL database support vendor and tried adjusting our configuration of various parts of the system to handle the large spikes in load we were seeing. However, we still were unable to identify the cause of the issue.

We gradually identified a cycle in the spikes, and began, one-by-one, to turn off internal loads that were periodic in nature (e.g., hourly database updates). Unfortunately, this did not reveal the problem either.

On the 29th of October, james_ logged in to a virtual machine that runs on one of our servers and noticed it felt sluggish. We then ran a small disc performance check, which showed severely degraded performance on this server. At this point, we realised that our assumption that the application was being delayed by a database problem was wrong. Instead, our web server was being held up by slow performance from the NAS, which is a different virtual machine that runs on the same server.

The NAS holds a relatively small amount of static files, including skin assets (such as background images), tag feeds, and work downloads (PDF, .epub, and .mobi files of works). Most page requests made to the Archive load some of these files, which are normally delivered very quickly. But because the system serving those assets was having problems, the requests were getting backed up until a point where a cascade of them timed out (causing the spikes and temporarily clearing the backed-up results).

To fix the issue, we had to get the NAS out of the system. The skin assets were immediately copied to local disc instead, and we put up a banner warning users that tag feeds and potentially downloads would need to be disabled. After tag feeds were disabled, the service became more stable, but there were further spikes. These were caused by our configuration management system erroneously returning the NAS to service after we disabled it.

After a brief discussion, AD&T and Systems agreed to temporarily move the shared drive to one of the front-end servers. This shared drive represents a single point of failure, however, which is undesirable, so we also agreed to reconfigure the Archive to remove this single point of failure within a few months.

Once the feeds and downloads were moved to the front-end server, the system became stable, and full functionality returned at 21:00 (UTC) on the 29th of October.

We still do not know the cause of the slowdown on the NAS. Because it is a virtual machine, our best guess is that the problem is with a broken disc on the underlying hardware, but we do not have access to the server itself to test. We do have a ticket open with the virtual machine server vendor.

The site was significantly affected for 118 hours and our analytics platform shows a drop of about 8% in page views for the duration of the issue, and that pages took significantly longer to deliver, meaning we were providing a reduced service during the whole time.

Lessons Learnt

  • Any single point of failure that still remains in our architecture must be redesigned.
  • We need to have enough spare capacity in servers so that in the case of hardware failure we can pull a server from a different function and have it have sufficient hardware to perform in its new role. For instance, we hadn't moved a system from a role as a resque worker into being a front-end machine because of worries about the machine's lack of SSD. We have ordered the upgrades for the two worker systems so they have SSDs so that this worry should not reoccur. Database servers are more problematic because of their cost. However, when the current systems are replaced, the old systems will become app servers, but could be returned to their old role in an emergency at reduced usage.
  • We are lacking centralised logging for servers. This would have sped up the diagnostic time.
  • Systems should have access to a small budget for miscellaneous items such as extra data center smart hands, above the two hours we already have in our contract. For example, a US$75 expense for this was only approved on October 26 at 9:30 UTC, 49 hours after it was requested. This forces Systems to have to work around such restrictions and wastes time.
  • We need to be able to occasionally consult specialized support. For instance, at one point we attempted to limit the number of requests by IP address, but this limit was applied to our firewall's IP address on one server but not the other. We would recommend buying support for nginx at US$1,500 per server per year.

Technologies to Consider

  • Investigate maxscale rather than haproxy for load balancing MySQL.
  • Investigate RabbitMQ as an alternative to resque. This makes sense when multiple servers need to take an action, e.g. cache invalidation.


Post Header

Banner by Erin of a spotlight on an OTW logo with the words 'Spotlight on Strategic Planning'

The Strategic Planning Committee is proud to announce that the 2016–2019 Strategic Plan has been approved by the Board of Directors and is ready for implementation! Strategic Planning is deeply grateful to the entire OTW for the patience, engagement and dedication they have shown throughout this process. The OTW would also like to thank our donors and supporters for making this possible.

Implementation of the plan will begin January 2016. We invite the public to peruse the Strategic Plan Visual Timeline to learn about the future of the OTW and its projects.

Organization for Transformative Works Strategic Plan Timeline

Stronger Infrastructure, icon created by Yamini Chandra for The Noun Project

Stronger Infrastructure

The OTW needs better technology, tools, processes, procedures, and documentation.

Stronger and More Efficient Board, icon by Till Teenck for The Noun Project

Stronger and More Efficient Board

A strong, democratic, and efficient Board will positively impact the entire organization.

Increased Volunteer Engagement, Retention, and Development, icon by Joao Santos for The Noun Project

Increased Volunteer Engagement, Retention, and Development

The OTW must build skills and experience in creating a stronger OTW-wide community.

Expand Fundraising and Financial Capacity, icon by Olivier Guin for The Noun Project

Expand Fundraising and Financial Capacity

The OTW requires funds to operate and grow, thus it must expand fundraising capacity.

Grow and Support Existing Projects, icon by XOXO for The Noun Project

Grow and Support Existing Projects

Each committee needs a clear understanding of their part in fulfilling the mission of the OTW.

Expand Fundraising and Financial Capacity icon by Olivier Guin for The Noun Project

Expand Inclusivity

The OTW celebrates the welcoming image it has achieved but must continue to promote representation and outreach.

6 Month Goals

Stronger Infrastructure, icon created by Yamini Chandra for The Noun Project

Document OTW-wide chair/lead position and expectations: Committee & workgroup chairs/leads work with the Board to draft a document which describes the organization-wide expectations for chairs and leads.

Preliminary purview statements: Committees/workgroups draft a statement, outlining their purview within the OTW including potential gaps or overlap between them and other teams.

Recruitment plan: Committees/workgroups document recruitment needs, training plans, timelines, etc. for new staff (and volunteers if applicable).

Stronger and More Efficient Board, icon by Till Teenck for The Noun Project

Purview and roles documentation: The Board drafts, approves, and internally publishes documents clarifying how the Board functions, in actually as opposed to ideally.

Phasing out and replacement of Board liaison structure: The Board implements flexible, clear, and structurally supported systems for staying in communication with committees & workgroups.

Expand Fundraising and Financial Capacity, icon by Olivier Guin for The Noun Project

Purview and roles documentation: As part of the Board documentation goal, the role of Treasurer is completely and accessibly documented.

Grow and Support Existing Projects, icon by XOXO for The Noun Project

1-year roadmaps for each project: Each OTW project will have a roadmap identifying the objectives and goals for the next year. This is: the Archive of Our Own, Fanlore, Legal Advocacy, Open Doors, Transformative Works and Cultures, Fanhackers, and the website.

9 Month Goals

Stronger Infrastructure, icon created by Yamini Chandra for The Noun Project

Document team-specific chair/lead position: Using the OTW-wide document outlining the roles and expectations of chairs and leads, all committees & workgroups draft their own document, adapted and elaborated for their specific leadership model.

Chair/lead succession plan: Committees/workgroups document a plan for recruiting, identifying, and training potential successors for the chair/lead position.

12 Month Goals

Stronger Infrastructure, icon created by Yamini Chandra for The Noun Project

Procedure for regular updates on committee/workgroup work: Committees/workgroups develop procedures for publishing via the internal wiki or other OTW tool the status of current work, including projects, and their plans until the next update.

Internal wiki pages for everyday and essential processes: Committee/workgroup creates and posts internal wiki pages that document their regular processes and procedures.

Purview overlaps addressed: The Board with committees & workgroups identify places where committee duties overlap and begin discussions to a) revise the committee duties to eliminate the overlap, or b) build a plan to accommodate the overlap.

Stronger and More Efficient Board, icon by Till Teenck for The Noun Project

Contested elections: The OTW prepares documentation and solicits Board candidates to be ready for a contested election conducted with transparency and voting by the membership.

Increased Volunteer Engagement, Retention, and Development, icon by Joao Santos for The Noun Project

Ombuds dispute system in place: The Ombuds dispute resolution system is approved by the Board and is fully implemented and running.

Committee-specific external skills training: The OTW identifies non-OTW resources useful for training staffers & volunteers in necessary skills for their specific committees & workgroups.

Expand Fundraising and Financial Capacity, icon by Olivier Guin for The Noun Project

Recruit second accountant/bookkeeper: The OTW recruits an accountant/bookkeeper to work with the current Treasurer.

Regular audits: A process is in place to conduct regular audits of the organization's finances.

Grow and Support Existing Projects, icon by XOXO for The Noun Project

Coalition guidelines: Legal develops a guide to the OTW's current coalitions with other organizations, and makes available on the internal wiki procedures for Legal to manage the creation of new coalitions and joint projects.

18 Month Goals

Stronger Infrastructure, icon created by Yamini Chandra for The Noun Project

Organization-wide processes for project management: Volunteers & Recruiting standardizes organization-wide processes for cross-committee project management and publishes process documentation on the internal wiki.

Mapping organizational structure: The OTW's organizational structure is easily describable and has been visually mapped.

Increased Volunteer Engagement, Retention, and Development, icon by Joao Santos for The Noun Project

Chair/lead basic training: Volunteers & Recruiting assists chairs & leads in developing and implementing training procedures for basic skills needed by all chairs/leads to be successful in the positions, such as leadership and management skills.

Grow and Support Existing Projects, icon by XOXO for The Noun Project

Revamp workflows and purviews for website: Clear and complete documentation exists regarding design, content management, support, and development.

24 Month Goals

Stronger Infrastructure, icon created by Yamini Chandra for The Noun Project

Committee/workgroup roadmaps: Committees/workgroups have developed a roadmap describing their priorities and setting specific objectives for the future, including potential new projects they may wish to undertake.

Committee/workgroup budgets: At least 3 committees/workgroups submit a budget to the Treasurer and receive feedback on it.

Stronger and More Efficient Board, icon by Till Teenck for The Noun Project

Operations structure with chair participation: The Board has revised its functioning and added new positions, structures, and procedures, so that OTW operations management incorporates participation from all committee chairs and current Board of Directors.

Increased Volunteer Engagement, Retention, and Development, icon by Joao Santos for The Noun Project

Skills labs/workshops: The OTW creates plans for skills labs/workshops in specific areas, such as wiki code or international outreach, and at least one is conducted.

Pilot mentors for chairs/leads: At let 4 chairs/leads volunteer their services and work with Volunteers & Recruiting to create a chair/lead mentoring program for the OTW.

Committee/workgroup plans for staff/volunteer engagement: Committees & workgroups create plans for regular check-ins, tracking progress, and rewards to show appreciation.

Grow and Support Existing Projects, icon by XOXO for The Noun Project

Fanlore-dedicated design & support staff: Fanlore acquires the needed staff to maintain and grow.

Roadmaps and/or accomplishments for each project: Each OTW project develops a public roadmap identifying its objectives and goals for the next 5 years. This is: the Archive of Our Own, Fanlore, Legal Advocacy, Open Doors, Transformative Works and Cultures, Fanhackers, and the website.

Expand Fundraising and Financial Capacity, icon by Olivier Guin for The Noun Project

Security overhaul: The process to update and improve the security of our financial and operational information and accounts is complete.

Committee/workgroup budgets: At least 3 committees/workgroups submit a budget to the Treasurer and receive feedback on it.

Expand Fundraising and Financial Capacity icon by Olivier Guin for The Noun Project

International and inclusive roadmaps: As part of infrastructure documentation goals, all committee roadmaps include explicit plans for inclusivity and international representation.

Inclusivity trainings: The OTW holds at least 3 trainings, open to all personnel, on internationalization and how individuals enact inclusivity and reach out to their communities.

Inclusivity survey: The OTW conducts at least 2 surveys for internal and external stakeholders regarding internationalization and inclusivity.

30 Month Goals

Stronger Infrastructure, icon created by Yamini Chandra for The Noun Project

Purviews clarified and documented: With Board's assistance, each committee/workgroup drafts, approves, and circulates a detailed document of their purview within the OTW.

Internal tool review and research: The OTW reviews current internal tools and researches better alternatives for suboptimal tools plus new tools to fit unmet needs.

Stronger and More Efficient Board, icon by Till Teenck for The Noun Project

Training plan for Board in place: A training process is available that can be utilized by all newly elected Board directors and by any qualified staffer who is considering stepping up to join the Board in the future.

Increased Volunteer Engagement, Retention, and Development, icon by Joao Santos for The Noun Project

Organizations-wide internal skills training: Mirroring the chair/lead basic training goal, basic training procedures are developed and implemented for skills needed by all personnel.

Complete chair/lead training plan: Expanding the basic training goal, the OTW implements a standard plan for all new chairs/leads covering both general and OTW-specific skills.

Administrative support/procedures for chairs/leads: Committees & workgroups create plans for handling routine communications, scheduling, minutes, task management, etc.

Expand Fundraising and Financial Capacity, icon by Olivier Guin for The Noun Project

Achieve 3-month reserves covering operating costs: The OTW has saved financial reserves sufficient to cover 3 months of the organization's operating costs without any additional fundraising.

Expand Fundraising and Financial Capacity icon by Olivier Guin for The Noun Project

Research best practices for international organizations: The OTW researches best practices for international organizations, prioritizing research on organizations that have a non-Western and/or non-Anglophone perspective and organizations that serve marginalized populations in the western, English-speaking world.

Survey data collected and reported: The diversity workgroup reports on the results of the analysis of the inclusivity surveys and prepares recommendations for the OTW base on said results.

36 Month Goals

Stronger Infrastructure, icon created by Yamini Chandra for The Noun Project

Organization handbook: The OTW creates and handbook which includes basic information on the OTW that every staff or volunteer needs to know, such as orientation to the committees & workgroups and how-tos for the organization's tools.

Purview gaps addressed: Where the purview statement have identified tasks being handled by committees & workgroups outside of their purview, this overextension is addressed.

Stronger and More Efficient Board, icon by Till Teenck for The Noun Project

Board succession plan: The OTW implements a Board succession plan where exiting Board directors & officers feel their knowledge and responsibilities are securely preserved and new Board directors & officers feel supported and informed in their training and transition.

New operations model plan: The OTW develops a plan for revising the operations management structure of the organization.

Increased Volunteer Engagement, Retention, and Development, icon by Joao Santos for The Noun Project

Organization-wide self-assessment: Self-assessment is implemented at every level: individual staff and volunteers, committees & workgroups, and the OTW as a whole.

Organization-wide external skills training: All personnel have access to external training programs and courses for skills that are difficult to develop internally.

Mentoring program for chairs/leads: The individuals available to mentor other in specific areas or skill sets is well documented on the internal wiki and the mentoring program is standardized across the OTW while remaining flexible to fit each committee/workgroup.

Explore a structure for inclusive annual meeting(s) that are not Board retreats: The Board explores the possibility of holding organization-wide meetings.

Expand and revise personnel outreach and recruiting strategies: The OTW expands methods an strategies for recruiting staff and volunteers.

Expand Fundraising and Financial Capacity, icon by Olivier Guin for The Noun Project

Annual fundraising raises annual operating costs plus 10%: The OTW's annual fundraising consistently raises enough to cover our annual operating budget plus 10%.

Expand Fundraising and Financial Capacity icon by Olivier Guin for The Noun Project

Inclusivity survey structure: A procedure, set of tools, documentation, and support structure are set up for the OTW to regularly survey internal and external stakeholders on issues of diversity, inclusion, representation, and internationalization.

All icons are from The Noun Project:

Stronger Infrastructure icon created by Yamini Chandra for The Noun Project
Stronger and More Efficient Board icon by Till Teenck for The Noun Project
Increased Volunteer Engagement, Retention, and Development icon by Joao Santos for The Noun Project
Expand Fundraising and Financial Capacity icon by Olivier Guin for The Noun Project
Grow and Support Existing Projects icon by XOXO for The Noun Project
Expand Inclusivity icon by OCHA Visualization Unit for The Noun Project


Pages Navigation