AO3 News

Post Header

Published:
2020-04-19 20:12:14 -0400
Translations:
Tags:

Several weeks ago, we took emergency measures to help the Archive handle an influx of traffic. These measures successfully reduced our database load, but left us unable to add any hits from logged out users to works' hit counts. We'll be able to resume counting these hits in the next few days, although there will be some changes to hit count behavior -- and one frequently requested feature we've been able to add thanks to these changes.

How hit counts will work going forward

Previously, a work's hit count only increased if:

  • you were not logged in as one of the work's creators, and
  • you visited a single-chapter work, a multi-chapter work in entire work mode, or the first chapter of a multi-chapter work in chapter-by-chapter mode, and
  • your IP address did not match that of the visitor right before you.

This meant hits weren't counted if you followed a direct link to a later chapter, e.g. from a subscription email, or if you were a work's sole dedicated follower who returned to it day after day.

The revised code will increase the work's hit count if:

  • you are not logged in as one of the work's creators, and
  • you visit a single-chapter work, a multi-chapter work in entire work mode, or any chapter of a multi-chapter work in chapter-by-chapter mode, and
  • your IP address has not visited the work in the last 24 hours.

This means a work's hit count will still only increase by one regardless of whether you visit one chapter or fifty, but it will no longer matter if you start on the first chapter or the tenth, or if someone else accessed the work between your daily visits.

As always, hit counts are updated approximately every half an hour to avoid placing unnecessary strain on our servers, although caching means it may take an hour for the new total to appear for logged out users.

Privacy

We don't want to hold on to your data any longer than absolutely necessary, so IP addresses used to calculate hits will be removed from our system within 48 hours of being collected.

However, the new code is JavaScript based, so you can disable JavaScript in your browser settings or by using a browser extension if you have any privacy concerns. (As always, our Privacy Policy has more information on what data we collect and how we use it.)

Hit count preferences

The preferences allowing logged in users to control the display of hit counts have been buggy or non-functional for years now, and the problem has only gotten worse as we've increased the amount of caching we do.

Therefore, we'll be removing the following preferences when the new hit count code is deployed:

  • Don't show me any hit counts
  • Don't show me hits on my works
  • Don't show other people hits on my works

If you'd prefer to avoid seeing hit counts after these preferences are removed, you can hide them with a site skin.

If you're not already using a site skin:

  1. Log in.
  2. Go to the Public Site Skins page.
  3. Choose "Create Site Skin" to go to the Create Site Skin page.
  4. Fill in the "Title" field with a unique title, e.g. "[Your username's] skin for hiding hits."
  5. In the large "CSS" field, enter the following: .stats .hits, .statistics .hits { display: none; }
  6. Press "Submit" to save your skin.
  7. Press "Use" to apply the skin.

If you're already using a public site skin like Reversi:

  1. Follow steps 1-5 from the instructions above.
  2. Press the "Advanced" section's "Show ↓" button to reveal more options.
  3. Press "Add parent skin" to add two new fields to the form, directly over the button. One field will be filled in with the number "1" and the other field will be blank.
  4. In the blank field, enter the name of the public site skin you are currently using. The autocomplete will help you find it.
  5. Press "Submit" to save your skin.
  6. Press "Use" to apply the skin.

If you're using a site skin that isn't public, you can try adding the code from these instructions directly to the skin, but it may need some adjustments depending on the other styles in your skin.

Latest chapter links

Now that going directly to any chapter of a work will increase its hit count, we've updated work blurbs to include a link that will take you from any work listing directly to the work's most recently posted chapter.

A work blurb showing the first 2 in Chapters: 2/2 is now the latest chapter link.

On multi-chapter works, the first number in the "Chapters" information at the bottom of the blurb will now be a link to that chapter, making it that much easier to get to the latest installment of your favorite work in progress.

We expect to roll out these changes in the next few days, once we've completed testing and coordinated our volunteers' schedules. We'll give a heads up on the @AO3_Status Twitter account shortly before we begin and add an update to this post once the changes are live.

Updated 08:30 UTC April 22: The changes in this post have now been deployed.

Comment

Post Header

The Archive has seen a marked uptick in traffic during March, with weekly page views increasing from 262 million to 298 million in just two weeks. We expect this trend to continue, and in order to keep the site running, we need to take emergency measures. The quickest, most helpful change we can make is caching the works we serve to logged out users. Unfortunately, this means that starting immediately, logged out users may experience a delay in work updates, and hits from logged out users will no longer be included in works' hit counts.

Why is this happening?

The increased traffic is putting a strain on our database servers, which receive dozens of requests every time someone loads a work. (We plan to order new hardware to help with this strain, but first we need to finish some ongoing server maintenance and determine which hardware to order. Delivery and installation of servers ordinarily takes a few months, and there may be unexpected delays due to the pandemic.)

Serving cached copies of works to logged out users will drastically reduce the number of database requests we make. Caching means we don't have to ask the database for the latest information every time someone visits a certain page in a given period of time. Instead, one of our front end servers gives everyone the exact same copy of that page. After about an hour, that copy is updated.

What changes will I notice?

Starting immediately, you may notice the following changes:

  1. When a new chapter is posted, logged out users will only be able to access it by direct link until the cache updates, which will happen about once every 60 minutes. Other changes to the work (e.g., edits made by the creator or new comments or kudos that have been left) may also not be visible to logged out users until the cache is updated.
  2. Because work pages need to be identical for all logged out users, we've had to stop automatically filling in guests' names and emails on the comment form. (You can request an invitation and create an account if you'd like the form to remember you!)
  3. Logged out users will see the adult content warning on every work rated either Mature, Explicit, or Not Rated. This is temporary and will be fixed as soon as possible.
  4. New hits from logged out users will not be added to works' hit counts. (Existing hits will not be lost.) The code that increases hit counts lives on our application servers, so it will not run when the front end servers hand out cached copies of works.

(This section was updated at 00:25 UTC April 1.)

Will hit counts be fixed?

We are exploring options that will allow us to resume counting hits from logged out users, but it may take some time to find and implement a viable long term solution. We'll work as quickly as we can, but we ask for your patience -- our volunteers may need to prioritize additional performance improvements or their own wellbeing in these stressful times.

We will keep you updated on any significant progress or setbacks here on AO3 News and on our @AO3_Status Twitter feed.

Updated 11:00 UTC April 24: We have deployed new code that allowed us to resume counting hits from logged out users, along with some general changes to how hits are being measured.

Comment

Post Header

We're making a few behind-the-scenes changes to kudos to make sure the Archive can continue to handle the tremendous number of them (659 million and counting!) that users and guests have left over the years. Kudos will continue to work the same, but there will be a new check in the database to make extra certain duplicate kudos can't be left. This will result in a one-time drop in kudos on works that already have duplicates.

Previously, we relied on the kudos code to make sure a single user or guest could only leave kudos on a given work one time. This worked just fine in most cases, but sometimes a slow connection or quick double press of the kudos button could lead to multiple kudos being recorded.

We'll be adding a constraint at the database level that should prevent this from happening. However, this change will also make any existing duplicates invalid and remove them from the database. This means that if you have received multiple kudos from the same user or guest on one of your works, your kudos total will go down when we deploy this change. For example, if your work has three kudos from the same user, the total number of kudos on the work will drop by two.

We plan to make our changes on Monday, March 16, and we expect some brief downtime at the end. We can't predict for certain what time it will be, but we will do our best to give a heads up on the @AO3_Status Twitter before it happens.

Comment

Post Header

Did you know you can post and browse works in over 70 languages on the Archive? In fact, AO3 is home to roughly 400,000 fanworks in languages other than English, with more posted every day! To help everyone find works in their preferred language, we've made some small changes to posting and searching.

Previously, the "Choose a language" field on the posting form was set to English by default, which made it easy to accidentally post a non-English work with the language set to English.

To fix this, we've updated the "Choose a language" field so English is no longer the default. Instead, everyone will need to select the correct language when posting a new work. If you forget, don't worry -- we've also added a friendly error message to remind you.

We've also made it a little easier to search or filter for works based on language! While we provide a "Language" field on our various search and filter forms, sometimes it's not enough -- for example, if you want works in either Spanish or Italian.

A search like that used to require knowing some numerical codes unique to the Archive. Now you can use these standard language codes with the search operators described in our cheatsheet. For example, to find works tagged with "Friendship" in either Italian or Spanish, you would enter language_id: es OR language_id: it in the "Search within results" field on the work listing for the "Friendship" tag. (The old numerical codes will continue to work, so there's no need to update bookmarks or links.)

We hope these changes will make the Archive a bit better for everyone, regardless of which languages you use for creating and consuming fanworks. And as always, if you notice a work with the wrong language, you can contact our Policy & Abuse team and they will help correct any mislabeled works.

Comment

Post Header

Published:
2019-11-07 03:36:14 -0500
Tags:

We're excited to announce we've recently added support for the <audio> and <video> HTML elements! With this change, you'll be able to use these tags to embed your self-hosted audio or video fanworks on AO3.

Unlike the Flash-based audio player we already offer, these new elements will work in all modern browsers, and they will continue to work even after Adobe ends support for Flash in 2020. (While we have no plans to remove Dewplayer at this time, we strongly recommend updating to the <audio> tag.)

Basics

There's still a lot of design, policy, and coding work to do before we can host audio and video files, so for now you'll need to upload your files to your own web space. Once you've done that, you can embed the file in your work with a small bit of HTML:

  • <audio src="AUDIO URL"></audio>
  • <video src="VIDEO URL"></video>

That's all it takes! Exactly how the resulting media player looks depends on the browser being used to access the work. We do, however, make sure that playback controls are available and adjust the width of videos using CSS to ensure big videos will fit on everyone's screens. We also make sure autoplay can't be enabled, and we add the preload attribute to gently suggest browsers save bandwidth by not loading the full file until you press play.

Complex examples

If you'd like to do something more complex, we support that as well. For example, you can include a poster for your video using the poster attribute (poster doesn't work for audio, but you can still include an image above your audio player):

<video src="VIDEO URL" poster="IMAGE URL"></video>

Because some older browsers don't support these elements, you can also include fallback text on either element to provide a download link:

  
<audio src="AUDIO URL">
  <p>Your browser doesn't support streaming with the HTML5 audio tag, but you can still <a href="URL">download this podfic</a>.</p>
</audio>

Because different browsers support different file formats, you might want to use the <source> element to include multiple formats.

  
<video>
  <source src="WEBM VIDEO URL" type="video/webm">
  <source src="MP4 VIDEO URL" type="video/mp4">
</video>

If you'd like to include captions or subtitles to improve the accessibility of your media file, you can do that with the <track> element:

  
<video>
  <source src="VIDEO URL" type="video/mp4">
  <track src="SPANISH SUBTITLE URL" kind="subtitles" srclang="es" label="Spanish" default>
  <track src="ENGLISH SUBTITLE URL" kind="subtitles" srclang="en" label="English">
</video>

List of allowed tags and attributes

Here is a full list of the tags we've added support for and which attributes you can use on them.

<video> element

  • class attribute
  • dir attribute
  • height attribute
  • loop attribute
  • muted attribute
  • poster attribute
  • src attribute
  • title attribute
  • width attribute

<audio> element

  • class attribute
  • dir attribute
  • loop attribute
  • muted attribute
  • src attribute
  • title attribute

<track> element

  • default attribute
  • kind attribute
  • label attribute
  • src attribute
  • srclang attribute

<source> element

  • src attribute
  • type attribute

You can get more information on using these elements and their attributes in these articles from MDN:

Edit November 8 at 07:53 UTC: If your audio or video file isn't loading on the Archive, you probably need to enable Cross-Origin Resource Sharing (CORS) on your website. Your web host's documentation or support department should be able to help.

Comment

Post Header

To cut down on abuse, we have just rolled out some changes to co-creating works, chapters, and series on the Archive. Until now, it was possible for any user to list you as a co-creator without your approval. We've added creator invitations to ensure you won't be listed as a co-creator until you accept the invitation. Additionally, we've added a preference that requires you to opt in before other users can invite you to become a co-creator.

Nothing will change about existing co-created items: all creators will still be able to edit the work, chapter, or series even if they do not enable the preference. Before you can be invited as a new co-creator, however, you will need to enable the "Allow others to invite me to be a co-creator" option on your Preferences page. (Where can I find my Preferences?)

Turning the preference off will not remove you from any co-created items, nor will it prevent any of the item's existing co-creators from making changes. You can turn the preference on and off as many times as you like. Turning the preference off will not delete any co-creator invitations -- you'll still be able to find them by following the "Creator Invitations" link on your Dashboard when you turn the preference back on.

If you need to add a co-creator who does not have the preference enabled and who you are unable to contact, you can use the Notes field to provide proper credit.

We're sorry we weren't able to announce this change ahead of time like we usually do, but adding co-creators who did not wish to be associated with a work was becoming an increasingly common abuse and spam tactic, and we didn't want to cause a spike in these incidents by publicizing it before this fix was live. We hope the changes provide relief for users who have been targeted and help everyone feel more in control of their experience on the Archive.

Comment

Post Header

Published:
2019-02-09 17:01:58 -0500
Tags:

Soon we'll be replacing our current method for generating work downloads with the Calibre command line interface. Our current patchwork system of third-party tools and custom code comes with a number of bugs and limitations, and moving to a well-maintained and documented application like Calibre will enable us to better maintain and even enhance the download feature going forward.

Because this is an entirely new system, there will be some differences between old and new files. We've provided an overview of these changes and some previews below, but we're still testing these changes, so there may be some minor differences between the previews and the final versions. Additionally, exactly how a file looks and behaves may vary depending on your device, software version, and settings.

HTML & PDF

HTML files serve as the basis for all of our download files, so it was necessary to make a few tweaks to help Calibre create tables of contents for other download types. Most of these tweaks are invisible, but we did make one visible change: the work title is now included at the top of the file, above the URL. Aside from these minor changes, HTML downloads will remain the same.

Because generating PDF downloads with Calibre would require us to install additional software on our servers, we will continue to use wkhtmltopdf to create PDFs, although we have upgraded it to a newer version. Like HTML downloads, PDFs will now include the title at the top of the first page but otherwise remain the same.

EPUB & MOBI

The biggest changes will be to MOBI and EPUB files, which are currently very different from each other due to limitations that existed when our system was created nearly ten years ago. The new code will give these files a more consistent look and feel and allow readers to take advantage of some features ebook readers provide.

EPUB

A current EPUB open in the iPhone Books app. The work meta is displayed in two columns inside a box. A new EPUB open in the iPhone Books app. The work meta is displayed as a bold heading, e.g. Rating, above the relevant tags.

The new EPUB files will do away with the two-column display of work meta (i.e. tags and statistics), which doesn't work well for small screens, lots of tags, or large font sizes.

This update will also fix the bug where EPUB files were missing chapter endnotes.

You can download an example of the new EPUB files to preview on your device.

MOBI

First page of a current MOBI file on the iPhone Kindle app, cropped to show the table of contents links provided at the top. A new MOBI file on the iPhone Kindle app, cropped to show the file's table of contents in the app menu.

Our current system provides a list of links on the first page as a makeshift table of contents, but Calibre will create files designed to work with your device or app's built-in table of contents function. (As you navigate with the shiny new table of contents, you may also notice each chapter begins on a fresh page.)

The text of a chapter in a current MOBI file on the iPhone Kindle app. The text of a chapter in a new MOBI file on the iPhone Kindle app.

Calibre will also add margins to prevent the text from running against the edge of the page. Additionally, we've switched from using indented book-style paragraphs to using vertical margins to create blank lines between paragraphs, bringing MOBI files in line with how text is formatted on the site and in other download files.

This update also brings a major improvement for internationalization: MOBI files of works written in non-Latin alphabets will no longer be automatically transliterated! Please note, however, that MOBI offers only limited support of right-to-left languages like Hebrew and Arabic, so Kindle users will need to use AZW3 to read works in those languages.

You can download an example of the new MOBI files to preview on your device.

AZW3

We'll also be adding a new download format: AZW3, which was created by Amazon for use with Kindle devices and apps. It's a newer format than MOBI, and it will allow us to offer Kindle users downloads of works in right-to-left languages, but it will otherwise look the same.

AZW3 downloads will be available on all works, but if you rely on Send to Kindle to transfer files to your device, you'll need to stick with a different format: Amazon doesn't allow you to send this file type. However, you can still transfer AZW3 files to your Kindle by downloading them to your computer and connecting your device.

If you're using the Kindle app on an Android device, you can move AZW3 files you've downloaded to your SD card's Kindle folder in order to access them. Unfortunately, there doesn't appear to be a workaround for the iOS Kindle app, so iPhone and iPad users will need to use another download format or a different, non-Kindle app.

Deploy date

While this upgrade won't immediately resolve all of the existing issues with downloads, it's an important part of the process, and it will ultimately make the Archive more secure and more maintainable. (After the deploy, our release notes will have more detail on specific bugs this change has fixed.)

We're still in the testing stages, but we plan to deploy these changes sometime after February 11. When the time comes, we'll announce it on our @AO3_Status Twitter account.

No downtime is expected, but if you have a work already loaded in your browser at the time of the update, you'll need to refresh the page to download it.

Comment

Post Header

🎁

Thanks to a long-planned database upgrade, the AO3 now lets you use all the latest emoji, as well as extended character sets for several languages (such as some of the rarer characters used for Chinese and Japanese) in your works, comments, or profile. 🎉🍾🎈🎊🎇. In fact, all text fields now support the UTF8MB4 character set, except for a few we've limited to Latin characters and some punctuation marks, like pseuds or collection names.

🔧

UTF8MB4 encoding only became available a few years after the Archive's inception, so we already had a pretty sizeable database using the older encoding when the upgrade became inevitable. After our recent server work, which included an upgrade to MySQL 5.7, people trying to post a work or comment with an unsupported character (e.g. one of the newer emoji) would be served an unhelpfully blank error page, exacerbating the problem.

However, migrating our entire 617GB database would have been a massive undertaking, likely necessitating several days of downtime. Fortunately, we were able to come up with a game plan that allowed us to keep downtime to a minimum by making changes to a new, empty copy of a database table while the "old" table with all the data was still in use. Once the new table was ready, we copied the data over and put the new table to use, then moved on to the next table.

😱

Thanks to the diligent efforts of our volunteer sysadmin james_, we completed the work in a little over a week and with no data lost. However, we ran into an unexpected problem with one of the constraints we put on our database to prevent the creation of multiple tags with identical names.

Due to a change in the way data is compared, some characters such as ss and ß are now treated as identical. That means that when we copied a tag like "Weiss Kreuz" to the new system, and then a few seconds later tried to copy "Weiß Kreuz," the second tag would be considered a duplicate and get quietly dropped. This made works originally posted to the dropped tag hard to find for a few days, since the tag didn't appear on the work anymore.

Once we discovered the issue and tracked down the cause, we were able to use a database backup to reconstruct the changes we'd made up to the point of the final swap, so we could compare the two tables and restore the lost tags.

👍

Together with our tag wranglers, we have put all the affected tags back into the system, so they will now appear on their works again. We are sorry if your works or bookmarks were affected by this issue!

Even though our upgrade is now complete and our database allows all of these new characters, keep in mind that some devices and browsers, as well as apps and e-readers relying on the MOBI format, might not display them properly. However, we hope you'll enjoy all your new emoji options and leave some extra-colorful love for creators in your comments! ❤🧡💛💚💙💜

Comment


Pages Navigation