AO3 News

AO3 performance issues

Published: 2012-06-01 02:38:41 -0400

As pretty much all of our users have no doubt noticed, we've been experiencing some problems with Archive loads: slowdowns and the appearance of the dreaded 502 page have become a regular occurrence. We're working on addressing these issues, but it's taking longer than we'd like, so we wanted to update you on what's going on.

Why the slowdowns?

Mostly because there's so much demand! The number of people reading and posting now is overwhelming - we're glad so many people want to be here, but sorry that the rapid expansion of the site is making it less functional than it should be.

We now get over a million and a half pageviews on an average day, often clustered at peak times in the evening (particularly when folks in the Western Hemisphere are home from work and school) - we were using a self-hosted analytics system to monitor site traffic, and we had to disable it because it was too overloaded to keep up. The traffic places high demands on our servers, and you see the 502 errors when the systems are getting more requests than they can handle. Ultimately we'll need to buy more servers to cope with rising demand, but there's ongoing work that we've done and need to continue to do to make our code more efficient. We've been working on long-term plans to improve our work and bookmark searching and browsing, since those are the pages that get the most traffic; right now, they present some challenges because they were designed and built when the site was much smaller. We've learned a lot about scaling over the years, but rewriting different areas of the code takes some time!

What are you doing to fix it?

Our Systems team are making some adjustments to our server setup and databases. Their first action was to increase the amount of tmp space for our MySQL database on the server - this has alleviated some of the worst problems, but doesn't really get at the underlying issues. They're continuing to investigate to see if there are additional adjustments we can make to the servers to help with the problems.

We're also actively working on the searching and browsing code: that's been a big project, and it will hopefully make a significant impact. Because it affects a lot of crucial areas of the site, we want to make sure we get everything right and do as much testing as we can to ensure that performance is where it needs to be before we release it. We're switching from the Sphinx search engine to elasticsearch, which can index new records more rapidly, allowing us to use that for filtering. That will offer us more flexibility, get rid of some of our slower SQL queries, and take some pressure off our main database, and it also has some nice sharding/scaling capabilities built in.

We also try to cache as much data as we can, and that's something we're always looking to improve on. Systems and AD&T have discussed different options there, and we'll be continuing to work on small improvements and see what larger ones we may be able to incorporate.

When will it be fixed?

It's going to take us a few weeks to get through all the changes that we need to make. Our next code deploy will probably be within the next week - that will include email bundling of subscription and kudos notifications, so that we can scale our sending of emails better as well. After that, we'll be able to dedicate our resources to testing the search and browsing changes, and we're hoping to have that out to everyone by the end of June. We rely on volunteer time for coding and testing, so we need to schedule our work for evenings and weekends for the most part, but we're highly motivated to resolve the current problems, and we'll do our best to get the changes out to you as soon as we can.

Improving the Archive is an ongoing task, and after we’ve made the changes to search and browse we’ll be continuing to work on other areas of the site to enable better scalability. We’re currently investigating the best options for developing the site going forward, including the possibility of paying for some training and/or expert advice to cover areas our existing volunteers don’t have much experience with. (If you have experience in these areas and time to work closely with our teams, we’d also welcome more volunteers!)

Thanks for your patience!

We know it's really annoying and frustrating when the site isn't working properly. We are working hard to fix it! We really appreciate the support of all our users. ♥

Comment

852 Prospect Import - Open House!

Published: 2012-05-23 02:58:23 -0400

As we recently announced, the much-loved Sentinel Adult Fiction Archive, 852 Prospect, is moving in with the AO3! You can find details about the move in our announcement post; however, if you have more questions about the move, or if you're just curious about Open Doors, you can join our Open Doors team for an open house this week.

The open house will take place on May 25, 2012 at 01:00:00 UTC (see when this is in your timezone) in our Public Discussion chatroom. It will last approximately one hour. You can access the chat at: https://fanarchive.campfirenow.com/eb4b6 - just click the link to drop by. Everyone welcome!

Comment

OTW Community Survey: First results

Published: 2012-05-09 10:11:40 -0400

The OTW Community Survey closed on Thursday 2nd May after its two-week run. An amazing 5986 people responded, which surpasses our wildest expectations. We'd like to thank everyone who took it and who helped to spread the word for making it such a huge success! We'd also like to apologise for the delay in posting about our preliminary results. We had originally intended for this post to go up the day after the survey closed, but unfortunately the mass of data involved and some unexpected commitments on the part of some key staffers meant that we weren't able to do this. We're sorry that we didn't post an update about the delay -- improving our processes to cover stuff like this is one of the things we're still working on. Thank you for your patience and your overwhelming response to the survey!

We wanted to cover lots of bases and get a broad picture of our userbases and their needs, which meant that this was a long survey! So we're especially pleased (and relieved!) that the completion rate was 82.2 % -- that is, 4921 people actually finished the entire survey and clicked the "done" button. (Don't worry if you were one of the other 18% -- we'll still be counting the results from partially completed responses.)

The high number of respondents corresponds with a humungous amount of data. Some of this will be reasonably easy to collate, while some of the free text responses will require more in-depth analysis. The final question ("Anything else you'd like to tell us?") alone yielded the equivalent of an epic Big Bang story: 60.000 words!

We've formed an internal workgroup who will help evaluate and publish the results. Due to the amount of data we're working with, this will be a gradual process, but we'll try to keep you up-to-date with the details as they emerge. We thank you for your patience while we get to grips with it all. The really meaty data may take us a while to collate, but here are a couple of tasty morsels to whet your appetite!

Where is everyone?

We had responses to the survey from fans all over the world - and a few from beyond this world (hello to the fan who responded from Narnia!) The top ten countries, by number of respondents, were as follows:

Pie chart showing percentage of survey respondents per country: USA - 56%, UK - 10%, Canada - 8%, Australia - 5%, Germany - 4%, Finland - 1%, New Zealand (Aotearoa) - 1%, Sweden - 1%, France - 1%, Ireland - 1%, Other - 12%

What types of fanworks are people enjoying?

Fandom is a diverse and wonderful place, and people are accessing a whole range of different types of fanwork. The vast majority of respondents (5613 of a total 5716) said they read fanfiction, but other types of fanwork such as fanart and fan videos of various kinds weren't far behind - and we can't wait to delve into the free-text responses to this question!

Bar chart showing numbers of respondents who said they consumed a particular type of fanwork: I read fanfiction - 5613, I watch fannish video (e.g. AMV, vids, machinima...) -	3856, I view fanart (e.g. drawings, digital art, manips...) - 4786, I listen to fannish audio (e.g. filk, podfic, soundtracks...) - 2552, I play fan-made game content (e.g. addons, mods, stand-alone games...)	- 778, No (I don't consume any fanworks) - 10, Other (please specify)	384

Stay tuned for more!

We're working hard on analysing the rest of the data and wrangling it into a nice accessible format. We really value all your input, and we'll do our best both to publicise it and respond to it. The response to the survey was beyond our wildest dreams -- we're feeling a teeny bit overwhelmed, but also excited about having such rich results to work with! Thank you to everyone who responded, publicised it, and worked to make it such a huge success.

Mirrored from an original post on the OTW blog. Find related news by viewing our tag cloud.

Comment

852 Prospect Moves In With the AO3

Published: 2012-05-08 18:29:45 -0400

Sentinel fans remember with nostalgia how Blair moved in with Jim. We have good news for those who know and love 852 Prospect, the Sentinel Adult Fiction Archive; it's moving in with the Archive of Our Own!

In this post:

  • a bit of background explanation
  • what this means for authors who have work on 852 Prospect
  • and what to do if you still have questions

Background explanation

The first iteration of the Automated Archive software was written by astolat for 852 Prospect back in 1997, but for some years that software has been growing increasingly creaky, and now requires all updates to be done by hand. As a result, alice ttlg, 852 Prospect's archivist, has been looking for a sustainable new home for the archive. That new home will be as an Integrated Collection in the AO3.

In its new home, 852 Prospect will be a separate, searchable collection with its own identity. We're currently exploring the possibility of doing a complete archive transfer, in which we could preserve old rec links by creating redirects at the original site. The archive will move in early June.

What does this mean for authors who have work on 852 Prospect?

If you have work on 852 Prospect, and you still have access to the email account associated with those stories, then when the import happens, you'll receive an email letting you know you can come and claim your works. If you already have an account on the Archive of Our Own under that address, you'll receive an e-mail letting you know that those stories have been added to your account.

If you don’t have an account on the Archive of Our Own, you will be invited to create one, and will be able to claim your imported works and associate them with that account.

All imported works are set, by default, to be viewable only by logged-in AO3 users. Once you claim your works, you can make them publicly-viewable if you choose. After 3 months, all unclaimed imported works will be open to all visitors, just like 852 Prospect currently is.

If you no longer have access to the email account associated with those stories, contact the Open Doors committee and we'll help you out. (If you've posted the stories elsewhere, or have an easy way to verify that they're yours, that's grand; if not, don't worry, we'll figure something out.)

If you have a story on 852 Prospect which you've already imported to the AO3, once the 852 Prospect import happens there will be two copies of the same story on the archive. This situation is temporary! Down the line, we’re planning to implement a way of merging two copies of the same work, so that you can keep your comments and kudos on the version you've already uploaded, *and* the redirect from the old archive address will still work. (Alternatively, we can change the redirect to point toward the copy you have already imported to AO3. Just claim the new copy of the work, and contact Open Doors with your AO3 account name(s), Prospect 852 account name(s), and links to the works that you want the redirect to point to.)

Once the stories from 852 Prospect are on the AO3, you can then edit, delete, etc, the same way you would with any work you posted on the AO3. Or, if you don't want the works to be part of 852 Prospect in its new home, you can choose to delete the imported works or to orphan the imported works. (Orphaning leaves the works on the Archive, but removes them from the control of any user account. You can choose whether to remove the name they were associated with on the old archive, or leave it displaying as a pseud.)

If you would like to prevent any works associated with a particular e-mail address from ever being imported into the Archive, please submit that e-mail address (and any info that can be used to verify your identity) to Open Doors.

If you still have questions...

If you have further questions, try the Open Doors FAQ; if the FAQ doesn't offer the answers you need, you can contact the Open Doors committee or leave a comment on this post and we'll respond as soon as we can.

We're also planning to hold two public chats on Campfire (the online chat platform the OTW uses): Sunday, May 19, at 18:00 UTC, and Thursday, May 24, at 01:00 UTC (click the links to see when the chats are held in your timezone). You can access OTW's public chatroom using this link.

Anyone who has questions about the process is welcome to join the Open Doors team at those chats, and we'll do our best to answer you.

Many of us in the OTW are longtime TS fans who really care about having these stories preserved for new generations. We're delighted to welcome 852 Prospect aboard the AO3!

-- The Open Doors team

Comment

Thank you for your support!

Published: 2012-04-26 17:11:14 -0400

Dear friends of the OTW,

Thank you. Your generosity over the last eight days has made this the most successful membership drive in OTW history.

Between April 18 and 25, we received 1,256 individual donations totaling US$35,744 — numbers that will grow once we've had a chance to count the donations sent via postal mail. On top of that, you successfully secured for us a $2,000 matching grant from our anonymous donor, bringing the current grand total for the drive to $37,744. And on top of that, more than 120 people have come forward to offer their time, energy, and skill as OTW volunteers (and 150 more have joined a waiting list).

These numbers are astounding, and they're all thanks to you. We are tremendously grateful to each and every one of you who donated, volunteered, or helped spread the word to make this drive such an enormous success.

Thank you, too, to all the OTW staffers and volunteers who supported the drive by writing and betaing posts, responding to donor inquiries, designing graphics, sending e-mail blasts, mitigating server traffic, monitoring donation receipts, contacting new volunteers, and myriad other tasks. This was truly an organization-wide effort.

Although this membership drive is over, it's never too late to contribute to the ongoing work of the OTW. In addition, our OTW Community Survey remains open through May 2 and is a great way for you to share your feedback with us. We also encourage you to check out our Strategic Planning Roadmap to learn about how we're working to enhance the long-term success and sustainability of the OTW and its projects.

Like all fan creations, the OTW relies on fans for its existence — and in the last eight days, your support has been inspiring. Because of your generosity, the OTW's efforts to preserve fanworks and fannish history, advocate for fans' legal rights, and promote positive representations of fans in academic and mainstream media discourse will continue to thrive.

On behalf of everyone at the OTW, we thank you.

Aja, Alex, Anna, Kat, Kristen, Lesann, and Liz
OTW Development & Membership Committee

Comment

Title - Tag Wrangling: It’s Your Right To Tag However You Like (You Can Even Be Your Own Spotlight)

At the recent April Showers Import Party, the topic of tagging came up. Unsurprisingly — there were fanworks finding homes on the Archive of Our Own, after all. And one of the most amazing things about the AO3 is definitely its tagging system.

What makes this particular tagging system so amazing? It's specifically designed so that users can use any tag, in exactly the form they want it on their works, while keeping those works as organized as they would be in a strictly classification-based archive. Perhaps even better organized, since AO3 tagging not only allows users to search for works using tags based on traditional classifications like fandom, character and relationship, but also tens of thousands of canonical "additional tags" that go far beyond the limits of genre. And the more you tag, the better it works overall (more on why that's so later).

So, you may be asking, what’s a canonical additional tag? And how does a tag come to be one?

First, a bit of tagging history is in order. The indexing of information using keywords isn’t a new practice, of course. When del.icio.us launched in 2003, the new part of their model wasn’t the link collections or the keywords themselves — those had been around on the Internet pretty much since the start. Their innovation was to give users the power to attach keywords to those link collections. Nearly ten years on, the AO3 has made a great start at putting that sort of descriptive tagging power in the hands of fan creators when it comes to archiving their fanworks.

A great deal of that power comes from additional tags (originally called freeform tags). They're tags that don’t fall into the standard fandom/character/relationship groupings, and include kinks, tropes, genres, story elements, word counts, recording lengths, video formats, fan art media, POVs, episode tags, additional warnings, and whatever else users can think of! And all those additional tags gain their useful descriptive power when they're made canonical, appearing in the search filters and the auto-complete box as the most useful, general forms of particular tags, with many other synonymous tags linked to them. Tag wranglers — fans who have volunteered to curate the tags belonging to particular fandoms — do the linking, so for a tag to have been made canonical means that a wrangler has either recognized it as complying with tag wrangling guidelines or created it in compliance with those guidelines specifically so they can attach another tag to it.

What that means is that when an additional tag appears in the archive, a tag wrangler assesses whether people searching for works would like to be able to search for works tagged with it. Often the answer is yes, but sometimes it's no — and that's fine! That doesn't mean that you shouldn't tag your works using whatever tags you like; those tags will still bring up works, after all, even if they aren't canonical! A tag wrangler making your tag canonical just means that they judged it a useful addition to the filters — and one that other people might want to have the option of tagging works with, as well.

So if you can tag with whatever you want, how do you effectively use additional tags to shine a spotlight on your work?

This is where we return to the question of how more tagging makes the system work better. You see, additional tags make it possible to bring up works tagged everything from Accidental Marriage to Zombies — and everything else in between. So, using tags which highlight tropes or kinks (among other things), means that people interested in these topics can more easily find your works containing them. And this isn't just limited to fanfic — there are additional tags which describe aspects of other types of fanworks that people might be looking for, as well — whether it’s a crochet pattern, a short podfic, or an example of digital fan art.

What it comes down to is this: your were-creature accidental marriage fic/podfic/painting/quilt with zombies and a female BAMF character of color lead might be exactly the work Fan X was looking for. And they might never know it exists, if you don't tell the world exactly what awesome stuff it contains. There're a lot of awesome fanworks featuring a lot of awesome stuff on the AO3 just waiting for people to spotlight those awesome qualities, so go forth and tell people that your fic is epistolary or your comic features a mythical being or creature or that your podfic is a cast recording.

While many of us love the diversity that additional tags bring, if you don't like seeing additional tags on works, you can always choose to ignore them, or even hide them completely using a custom skin and the Blurblings Hide freeform tags skin.

If you'd like to know more about tagging on the AO3, or about what tag wranglers do, the Tag Wrangling Committee is having an open house on Sunday, April 22, from 19:00 to 21:00 UTC (see when this is in your timezone) in OTW's public chatroom on Campfire. The chatroom can be accessed at: https://fanarchive.campfirenow.com/c6440 Feel free to drop by at any time during the session to ask questions or just to hang out.

The AO3 and its tagging system are funded by fans, for fans. To help keep it growing, please donate today.

Comment

Release notes for release 0.8.13

Published: 2012-04-21 13:13:14 -0400

Welcome to Release 0.8.13. Elz, Enigel, Firewolf, mumble and Sarken contributed code to this release, which was tested by our awesome testing team: Enigel, Jenn, Kylie, Sarken, mumble, Xparrot, and Zebra.

This early release is brought to you courtesy of a bug which was introduced in the last release. Trying to post a draft without previewing would only save changes to the draft, not publish the work. Unfortunately, it interacted with other draft-related bugs and resulted in complications for several users. We're choosing to deploy fixes for these bugs early, and move other changes planned for April to the May release instead.

Highlights!

More informative page titles!

Did you ever bookmark a fandom or pairing tag only to have it show up in your Delicious or Pinboard account as "works index" without any information whatsoever? Do you navigate to a page and view the title only to see something like "Archive of Our Own » users show" and feel bemused? Bemusify no more, page titles have been added that are more meaningful! Most pages now display more granular information which will hopefully make for less bewildering bookmarks and easier browsing; some work remains to be done though. Please bear with us while we make sure that all pages have helpful titles and do feel free to point out anything that seems off to you.

Series subscriptions!

In response to popular demand and following subscribe to user and subscribe to work in progress, we have added subscribe to series so you can keep up with those lovely, long series. You can find the Subscribe button at the top of all series pages. The current release will not trigger an email notification if an already-posted work is added to a series, however. This will be fixed in the next code push.

Note to authors: Your series will show up as a work in progress until you edit the series information and tick the "This series is complete" box at the bottom.

Known Issues

See our Known Issues page for current issues.

Release Details

Features

  • Page titles have been updated to show more meaningful information!
  • You can now subscribe to a series as well as users and works! \o/
  • Tag wranglers can now mark and manage tags as unwrangleable!

Bug fixes

  • Works
    • The character limit for chapters was interacting in unfortunate ways with the paragraph tags added by the parser, resulting in invalid works (that couldn't be bookmarked, for instance); this has been fixed by adding a buffer for HTML tags to the allowed chapter length
    • Trying to post a draft without preview wouldn't actually post the work, resulting in a work that only looked published; this has been fixed
    • The saved draft of a work would appear exactly like a published work to the author, potentially causing some confusion; a small notice banner at the the top now informs you that the work hasn't been posted yet
    • When commenting in view-full-work mode on a work with unposted chapters, the comment would get "attached" to the last chapter, even if it was still a draft; this has been fixed
    • On a pseud's work index, it would say "# Works found by USER", even though it was only a subset of works by that user; it now says "# Works found by PSEUD (USER)"
  • Series
    • The option to add a co-author to a series was broken, making it impossible to have a shared series with another user; this has been fixed now
    • Trying to edit a series and going over the character limit in the Description or Note field would result in an error 500, this has been fixed
    • Before deleting series information from a work, the confirmation message would only say, "Are you sure?" which wasn't very clear; it will now clarify that the series will be deleted from all works (not just removed from this one)
    • When adding a new series to a work and using "Post without preview", the series would become visible to logged-in users only with no way for the author to lift that restriction; this bug has been fixed and only series consisting entirely of restricted works will themselves be restricted
  • Frontend
    • Some pages, such as the Donation page and the FAQs were hard to scan; whitespace and formatting have been adjusted for easier processing
    • There were instances of text-transform and font-variant in our stylesheets that were deemed problematic from a translation perspective; these have been removed and all user content and site elements should now show up as intended
    • Some sections on several Archive pages didn't adhere to our standards for CSS classes, making them difficult to style in skins; those have been tidied up
    • The draft notice text on chapter drafts was obscured by a wonky line; this has been fixed
    • The explanatory text for the "Canonical" field on tag wrangling pages was missing a paragraph tag; it has been added now
    • The homepage was missing some landmarks (important for screenreader access); they have now been added
    • The stylesheet had an invalid cursor value for links; this has been fixed
    • The stylesheet links in the site header did not meet XHTML Strict standards as the tags weren't closed; they are now
    • Editing a comment and going over the character limit, the text field would shrink when clicking Update, making it difficult to edit the comment down to the proper length; this has been fixed
    • In the filter box on work pages, the arrow and filter names now trigger the pointer cursor, making it more obvious that they can be clicked to expand
    • The "hidden by admin" icon would not show up in the respective bookmark blurbs and help text box; this has been fixed
  • Misc.
    • Previously it was only possible to add alt text and comments to a user icon; the same can now be done for a Collection icon
    • The edit/wrangle/works links on tag wrangle pages were set to open in a new window or tab; this has been changed so they open in the same window now
    • The text in password reset/generated password emails was missing a word; it has been added now
    • There was a small HTML error in gift notification emails; this has been tidied up
    • When creating an account and repeating the chosen password wrong, the resulting error message would be wonky in several ways; this has been fixed now
    • The deviantART importer (using expanded fav links, e.g. URLs in the form of /art/TITLE-12345) has been fixed to work with recent HTML changes on dA's end

Comment

Let's ticky! OTW Community Survey

Published: 2012-04-20 05:17:08 -0400

"What language(s) do you use in fannish or fandom contexts?" "Have you ever used the Fanlore Wiki?" "How did you first hear about the OTW?"

These are questions from the OTW Community Survey, a massive ticky extravaganza designed for one purpose: to find out how we can serve you better. It includes questions on AO3 and the other projects of our parent organization, the Organization for Transformative Works.

The members of Internationalization & Outreach have spent most of the last year gathering ideas for questions from the OTW's many committees. Our goal was to design the kind of survey we ourselves wouldn't mind taking: mindful of fans' need for privacy, conscious of the international nature of fandom, defined in scope, and free from frustrating limits on answer options.

We want you to tell us what you think about our projects and the organization in general, where you think we're succeeding or failing. And we also want to learn more about who you are. We're excited to finally open the doors and invite you in. Please bring your friends and help us make the survey a success by spreading the word in your communities!

The OTW Community Survey runs through May 2nd and is hosted on a secure connection at Surveymonkey:

https://de.surveymonkey.com/s/surveyOTW

OTW Community Survey, April 18-May 2, 2012, survey URL

OTW Community Survey

And if you're as curious as we are about the results: we will be publishing them in aggregate sometime after the survey closes on May 2nd. If you have any concerns or questions in the meanwhile, please contact the Internationalization and Outreach Committee!

Comment


Pages Navigation