Thursday, May 26, 2011

How ePub came to Kindle-land (a Fairy Tale)

Once upon a time, a humble bookseller named Bezos had a compelling dream: "Every book ever printed, in any language, available to read in under sixty seconds."

With these words still echoing in his head upon awakening, he looked around. There were some people reading books on their desktop and laptop computers. A few on their Palm Pilots and Pocket PCs and Symbian phones. There were seemingly dozens of digital formats, but limited availability of professionally produced content. This type of reading was an unpleasant and inconvenient and dangerous activity, undertaken only by People of the Fringe, willing to withstand the rigors of mobile, digital reading without complaint or concern for their deteriorating eyesight, due to the constant eyestrain caused by reading miniscule text on low-resolution, backlit computer displays. Bezos cried out to the skies in despair: how could he ever realize his wonderful vision? He was, after all, just one man. (Well, okay: a very rich and powerful man, with billions of shareholder dollars at his disposal.)

So he looked around some more. There was a new, magic, low-energy reflective display called 'e-ink', that had a paper-like look to it. Even ordinary people could stare at it for hours and hours and hours without the slightest discomfort. There was even a new device designed to utilize the Magic Screen for the specific purpose of reading digital text, confusingly called 'Sony Reader' (the device itself could not actually read anything, it was up to the user to do the reading). He also learned that Sony didn't understand publishing, or marketing, or people who read books, or even where books come from, and that this was just another piece of electronics to them. "They don't understand," Bezos thought. "I guess I'm just going to have to make one of those things myself somehow."

He looked further, and wider, to the shores of Europe, and finally, to Paris. There he met Monsieur Mobipocket (an unusual French name to be sure), who had developed a Reading System that worked on many mobile devices, coupled with a small but vibrant internet bookstore, with a number of participating publishers. For, it is true, the Reading System included the semblance of protection from the ravages of Digital Piracy, as demanded by said publishers. The System used its very own Format, adopted from ideas that had taken shape as the Open eBook Standard. They named the Format 'Mobi'. Mobi was very small, and very quick, and could render formatted text on the smallest devices using but the puniest of computing resources. But these devices were the same devices that only People of the Fringe could understand or tolerate. Bezos liked what he saw, but something was missing.

Following his meeting with M. Mobipocket, Bezos' mind reeled with possibilities. What if...? Yes, what if you had a Magic Screen device, added the wireless connectivity of a mobile phone, and combined it with M. Mobipocket's System? He did some quick calculations on a napkin at what had become his favorite café in Paris, "Le Kindle", withdrew a tiny portion of the money which his Investors had given him, and set immediately to work.

He rushed back, cash in hand, to buy M. Mobipocket's System. Their catalog provided the nucleus of a new, rapidly expanding one, as publishers got even more content in quickly with the help of Bezos' ebook-fabricating dwarves. He had some of his elves design a Sony Reader clone with a Magic Screen, but with a keyboard, for he knew readers would want to add their own notes to the books they love, and the touch screen technology of the time would have reduced the Magical properties of the Magic Screen to almost nothing. Some other elves adapted the System for wireless delivery.

Finally it was finished. He named it 'Kindle', priced it just below Sony Reader's price. Homely as it was, it quickly became a best seller and it was time to refine and improve the System further. The elves added TTS, a web browser, and fairy dust. An update added PDF support, with it, Adobe's Reader Mobile SDK. And sharper fonts. Always the fonts must get sharper.

"Wait. Isn't that the very same Adobe RMSDK that enables rendering of ePub files, and support for DRM that allows sharing of content among the reading systems that license it?" you ask. Why yes indeed. We were just about to introduce Mobi's younger sister, ePub, and you'll come to understand everything about this.

ePub, like most younger siblings, learned by example and improved on it. She refined Mobi's more primitive formatting. Adobe had known her since she was born, and incorporated her into their expanding digital publishing business. Soon they partnered with Sony and developed a mobile reader SDK to rule them all, combining the new digital document standard, ePub, with Adobe's old digital document standard, PDF, so that anyone could create a reading system quickly with all of the basic functionality required. Combined with Adobe Content Server, it comprised a secure ebook delivery system like that which M. Mobipocket had pioneered.

Even as pleased as Bezos was with Kindle, he recognized the importance of PDF to many readers, and while Mobi was proving its worth, in the back of his mind, Bezos understood that support of standards like ePub was important, and that one day, he would want Kindle to work with her as well. But there was so much to do, and it was all so exciting! and so very Profitable! So he licensed the Adobe RMSDK, had the elves implement the basic PDF support that was required, and moved on to other things.

There was a larger screen Kindle. Then a new Kindle with an even more Magical Screen, wi-fi capability that provided a lower cost option with wireless capability, Collections, social networking features, even sharper fonts, basic support for Cyrillic, Japanese, Chinese, and Korean script, and even Real Page Numbers! Kindle apps for Android, iOS, Blackberry, Windows Mobile, Windows, and Mac, all in perfect synchrony! A more affordable, Kindle with Special Offers! Library borrowing capability was announced. Was there anything more to be done?

It was at this point that the ePub3 specification was released. ePub was all Grown Up! All the ebook designers wanted to work with her, and explore her many fine qualities and capabilities. Mobi was still functioning well, but his time was nearing an end. Reading Systems, including Kindle, were gaining more power, and no longer required or even appreciated his elegance and efficiency. Even Bezos was smitten with ePub. It was time to bring her on board.

He began talking with his elves and marketing fairies and business wizards. "How can we make this happen? How can we help Mobi have a well-deserved retirement? How can we make ePub happy here?"

The elves spoke first. "We already have the Adobe RMSDK. We didn't dare tell you before, but we already have a beta version of the firmware modification that will allow the current Kindle models, and even the penultimate generation models, to render ePub handily and seamlessly. We can continue to make Mobi files for the Kindles and Kindle apps that can't read ePub. A large portion of the source we get is already in ePub format, and we've been setting these aside just in case and have tested everything on our development servers. We're really smart, in case you didn't notice." If only they paid us accordingly, they mumbled under their breath.

The business wizards interjected. "That's going to be expensive! MAYBE we can update the current models, but not anything before that! And there's no way we're going to license the Adobe server and pay them transaction fees! Jobs saw right through that scam!"

The elves responded. "Well as to that, there's really no technical issue. We can handle multiple formats, and deliver the appropriate format to the appropriate reading system, we created all of them and know their capabilities. We'll use the existing device IDs to generate the encryption keys to apply DRM to the ePub content, as per the specification. We don't need the Adobe server for any of this, we're only selling these to Kindle customers."

The marketing fairies could no longer contain themselves. "We want everyone to be a customer! We want to sell ebooks to anyone no matter what reading system they are using! They will come to appreciate the advantages of becoming Kindle customers as well, with all of the exclusive content and services we offer, and we'll need to let them take the ebook content they already own with them, or they'll never switch to us. We have got to support Adobe DRM."

The wizards hemmed and hawed. "Well, I can see your point. We can always use more customers, and we won't actually be giving non-Kindle customers the services they would have as Kindle customers. We can probably afford to pay Adobe a little something for that side of the business. But what about the rest?"

One of the elves, who was something of a smart-ass, spoke up. "You do realize that in the Kobo reading system, Adobe only gets paid for licensing fees of the RMSDK that is used on their reading devices to allow side-loading of 3rd party DRM content—it is not licensed for their reading apps, which use a non-Adobe ePub rendering system—and Adobe DRM is applied by Kobo's Adobe Content Server only when someone requests to download an ePub file for reading on a non-Kobo reading system? You do realize that's really all we would be doing? Anybody can see that, can't they?"

Bezos weighed in. "Okay, enough of the attitude, Elvis, but that makes it pretty clear. How soon can we do this? I want to be first out the door with an ePub3 implementation. Work out the arrangements with Adobe, have Legal go over things to see what we need to tie up with publishers and get them to give us their ePub files ASAP—that is, the ones we don't already have because they were to darn lazy to convert them themselves. It's time to join IPDF and get ePub to like us! To like us a lot!"

And so it came to pass. And everyone lived happily ever after. And every book ever printed, and even those never previously printed, in every language, was ready to read in under sixty seconds. (*)


(*) With a fast internet connection. When they don't have embedded audio and video. Statement assumes people still read and write.

Wednesday, May 4, 2011

Kindle does Libraries!

Amazon recently announced that they had completed a partnership with leading library vendor, Overdrive, to offer titles in Kindle format through Overdrive's many public library customers. Even as confident as I had been that such a deal would take place at some point this year, I found myself surprised and thrilled to learn that it was indeed moving forward.

As a denizen of many a Kindle discussion forum, I've witness many a thread complaining that they could not borrow ebooks from the public libraries that their taxes support, despite Amazon's dominating market share (in the U.S.A. at least) in ebook sales and in dedicated ereading devices. The usual discussion ran as follows:
"Why doesn't Amazon make it possible to borrow books from the library like Sony and Barnes and Noble do?"
"Because their priority is selling books. If you could borrow a book, why would you buy one from them?"
"The selection at libraries is lousy, and the wait lists are long. You aren't missing anything."
"The publishers don't want to give Amazon more market share and leverage than they already enjoy. They want to make sure ePub remains a counterbalance."
"Overdrive has decided Adobe ePub is all they need. They don't want Yet Another Format."
"You idiot. You knew what you were getting when you purchased a Kindle, and it didn't include the capability to borrow ebooks from your library. Get over it already. Go buy a Nook and leave us alone."

I tried to argue to the contrary:
Given the inherent scarcity of content borrowed from libraries, Amazon would much rather the borrowing took place on a Kindle than on a competing device, because any ebook purchasing a Kindle owner does will most likely be with Amazon rather than a competitor.
Libraries know that most of their patrons have Kindles and want to borrow ebooks to read on them.
Overdrive wants to be the company that libraries turn to for fulfilling demand for ebooks. If Amazon were to partner with one of their competitors, it could seriously disrupt their business. But if they partner with Amazon, when none of their competitors do, they become even more indispensible.
Furthermore, Overdrive has already has demonstrated that they have the technical ability to deliver Kindle format. They have been distributing Mobipocket format for much longer than Adobe ePub, Amazon owns Mobipocket, and the format and DRM system is virtually identical.
Publishers can't afford to ignore Amazon customers or the potential library market. They make significant amounts of money on library sales, and when libraries cannot fulfill demand, it drives purchases in the end. They may not always like dealing with Amazon, but they have no choice when Amazon sells so many of their books.

And so it has transpired.

I was concerned that any such agreement would force libraries into taking sides in the ebook format war, and have to purchase the same titles twice. However apparently that's not the case: Overdrive can offer the same title in any available format for the same price. Publishers get paid the same regardless of format. Patrons get to choose the one they want. Adobe and Amazon split the DRM licensing revenue, depending on the relative popularity of their respective solutions.

Needless to say, demand for library ebooks, already skyrocketing, is going to spike when Kindle format comes on line. But there's no holding back the tide: library patrons have a right to expect their library to deliver media in all popular formats, and while ePub is popular, Kindle format is currently even more popular.

I'm very curious about how this will be implemented. Interestingly no software updates of Kindle or Kindle apps will be required: this capability has long been demonstrated (since Kindle 1) by a simple hack of Mobipocket ebooks to make them readable on Kindle without stripping DRM, and such books expire at the end of the lending period.

However, the server side of things will need to change. To borrow a Mobipocket ebook, you have to supply a 'PID' (device identifier). The Kindle platform involves PID's as well, but there is no way for the average user to discover what it is for any given Kindle or Kindle app. So I suspect that after choosing a book to borrow on the library's Overdrive site, patrons will be redirected to an Amazon site to authenticate with their Amazon credentials, choose the device to prepare bits for, and the delivery method (download to computer or use 'whispernet'). I think the entire fulfillment back-end will be Amazon-hosted, not Overdrive-hosted. All that Overdrive's site will need are the ASIN's (Amazon item id) for the titles it offers, so that it can supply this when redirecting to the Amazon site.

With the addition of library borrowing, Amazon has erased one of the last feature advantages competitors had. While the lack of this feature didn't seem to impede Amazon's success, it no longer represents a consideration for those contemplating a purchase decision.