Amazon Web Services Chat 21 June 2007

The AWS chat is getting interesting. Today there were some useful tips. I learned that with libsocondlife it is possible to upload textures on to SL without the L$10 tax! Cool.
Connecting with other Mturk enthusiasts was an added bonus.

It is a good forum to make feature requests and connect with other developers and listen to their ideas, concerns and have fun.
Today Ryan Mahoski/Drunsen Deadlight brainstormed about a audio wikipedia built based on Mturk and telephone based service 1-900-WIKIPEDIA. [ Initially it started as 1-800-WIKIPEDIA, then it evolved into 1-900-WIKIPEDIA – permium telephoney service to charge the end user] Interesting!

I requested a Mturk sandbox feature: Replicate the qualifications from the main Mturk on to the sandbox so that it is easy to test with qualifications.


About labsji

I blog, Therefore I exist ! Funny things are funny to me. Cool things are cool to me. Innovations tick me. I attempt spirituality religiously :)
This entry was posted in Amazon Web Service, mturk, Second Life, Secondlife,, serindipity, unconference. Bookmark the permalink.

4 Responses to Amazon Web Services Chat 21 June 2007

  1. Ryan Mahoski says:

    Thanks Balaji, I’m glad my idea of an Mturk driven audiopedia resonated. Xpert and Jeffronius deserve credit for developing that hypothesis into a compelling VoiceXML, podcast-ready, solvent use case. The collective creativity for affecting education, to say nothing of your own contributions from our brainstorming session, was inspiring.

    I thought your feature request was perceptive and raised an interesting issue. If I understand you correctly: qualifications created on the live site should ideally propagate to the sandbox–automatically, in order that the two may be in sync. I have not tinkered much with qualifications in the sandbox, which is probably why I don’t yet see the full value of your enhancement. It seems to me, if qualifications on the live site drive qualifications in the sandbox then any qualification edits one makes to the sandbox will be immediately overwritten by the live site. I can appreciate the value of syncing the two (saves time), but wonder if this enhancement would mean we could no longer test qualifications. In case you happen to be developing in Ruby, I wonder if the Rake utility would help you propagate your qualifications schema? Else, I notice in the forum that Randy helped Joanna solve a (Java, sandbox) qualifications issue–maybe post there if not just to solicit comments from the larger Mturk dev community.

    Re: 1-x00-WIKIPEDIA I should begin by saying I am not the first to suggest an audio version of Wikipedia. Last year a small but determined group of Wikimedians began narrating articles. Their effort has been interesting but the impotent quality and volume of that library indicates their process is probably unworkable.

    The problem is, they adopted the same collaboration setup for multimedia as Wikipedia uses for text. Threaded chat works in Wikipedia but for audio-video textual discourse stifles important dialogue about form and polish. If we agree that narrators should be scripted and that a narrator’s script is the Wikipedia article, then the journalism piece is necessarily complete. Any discussion about quality becomes one of artistic performance and elegance.

    Is it possible to judge form objectively? On some metrics yes, for example Paul Graham writes lucidly about timeless art in Hackers and Painters. But judging from a Turing test of today’s text-to-speech software, we barely have a framework/theory/model from which to begin the debate. Multimedia evaluation is mostly subjective.

    One solution is to remove the interlocutors and invite listeners to decide the quality of audio (and later, video). In a sense popularity, not neutrality, should drive the porting of text to multimedia. This suggests our feedback mechanism ought to change from one driven by altruism to a socio-financial incentive structure.

    I do think this is one of myriad opportunities using Mturk–with its rational system of qualifications and bonuses–to triangulate and reward value. In the case of audiopedia, one way to connect incentives to mass media production might be standard ad revenue sharing. But as Jeff said, this setup is tantalizing for those who would seek to game the system. I can design an algorithm to police villains but the better solution may be to table audiopedia for now in lieu of a less lofty but solvent project.

    As with all internet business, the ideal is not AdSense but rather services for which people are willing to pay directly. The possibilities are too numerous to list, but in keeping with the current narrative let us just say that workers with voice aplomb might find it convenient to read blogs aloud for the busy writers who seek a greater audience size. If producing podcasts for bloggers doesn’t sound profitable enough, consider the budding market for vlog post-production. Or what about video production writ large, as in “All workers are hereby qualified to join a self-governing, transparent (save your identities) organization / The requester designed only some programmatic framework and the mission statement ‘Produce the daily news’ / Triangulate for yourselves each day the requisite script, on-air talent, pre- and post-production objects / Your budget (i.e., rewards + bonuses) is limited only by nominal variable costs (AWS fees + nominal percentage for the org’s humble founder) / Sales channels including Revver and Google AdSense provide transparent metrics and payments which, along with your peers’ assessment of your artistic contribution, your rewards and bonuses are determined / This is an iterative routine and unlike most Mturk HIT Groups, this one does not “cap” your income potential.”

    I can’t help but digress about the potential for other industries. It’s conceivable that one could create in Mturk a complement to or a replacement of oDesk, to use a software example. I personally would love to create qualifications for workers who have a bent for medicine–so that, in the event I get sick, I can request this cadre of hack physicians to quickly review my symptoms, medical records, and imagery–not to replace my local doc but as a credible research tool, an ad-hoc triage unit. Is this considered “practicing without a license?” I honestly don’t know but to find out, maybe I could save myself attorney fees by creating an Mturk HIT to counsel me in light of the applicable statutes (I’ve actually requested something similar in Mturk, with great success–workers even cited all their legal sources.)

    I’m sure if we brainstormed a little more we’d find there’s more to add to these apps than just VoiceXML, billing support and social promotion. A coder with scant ingenuity can design a viable revenue model, build the UI, and add features to taste. Each iteration in Mturk costs serious money, so you should always be trying to minimize AWS’ cut. I can think of some interesting ways to do that but of course we don’t increase revenue by cutting costs, so the answer is to do both.

    I think with Mturk, some of the quaintest hypotheticals can be viable businesses. Naturally, somebody has to code the logic and because coding is far more difficult than brainstorming I suppose it is not surprising why no one is building remarkable HITs. In other words, we don’t even seem to have remarkable ideas yet. It doesn’t take much insight, however, to see the potential use cases. So what gives? With clear scripting and production notes, like Alfred Hitchcock a good coder can sleep through the performance.

    Apart from our dev chat, what have you heard? I subscribe to a lot of search feeds and, from what I can tell, the discussion on Mturk development is nil. The only voice is that of workers complaining about uninspired HITs. Am I the only one who is drinking the Kool-Aid? I realize the 10% tax is significant, but if even if AWS increased it to 15% it seems the break-even point for these use cases is around 5% tops.

    – Ryan

  2. labsji says:

    That is lot of Mturk Kool-Aid. Heartening to note that there is someone else who has drunk more Mtrik Kool-Aid than me.

    Mturk is simply brilliant so it takes a while to get factored in. It does not take a lot of imagination to stuff your bits onto S3 – need is there and deal is good, so it is a overnight success.

    Whereas programming a HIT involves taking care of ingenious work evaders – humans. Indeed, modern science and the world around us is centered around this aspect. Developers just get bogged down by it. At least I got bogged down by it.

    Mturk Qualifications are the crucial piece to counter the human work-evasiveness. As Mturk is in its early stages, there are not enough cookie-cutter qualifications, not to mention enough workers holding the qualifications. I want to play around with qualifications, hence the request for replication from the main grid. Qualifications cannot be deleted from the main grid is a deterrent for not playing with qualifications in the main grid. [But there is a major risk for Amazon if they start replicating the main grid in sandbox – eventually some people will start using the sandbox to bypass the 10% mturk charges – talk about human evasiveness!]

    Concept selling Mturk is like concept selling Wikipedia before Wikipedia got proven. I have failed in couple of my attempts( actually yet to see success – I have not given up). I’m taking the game route to prove the first use case. SL is the way to go. I’ll keep you posted and look forward to your feedback and support.


  3. Ryan Mahoski says:

    “Really we were just _playing_ in the sandbox. Honest!”

    When I said always try to minimize requester fees, I didn’t realize you could totally solve the problem, Labsji. Well done. Bonus granted if you can figure out how to pay the worker.

    I agree qualifications are important, but I also think it’s advantageous to create hybrid HITs. By hybrid I mean open to both qualified and unqualified workers. I trust a qualified worker more than I do an unqualified worker but, to an extent, that just means I don’t have to check/triangulate the former and can afford to pass along the savings. It would seem best to grant qualifications based, not on some game-able test, but rather on past HIT performance. I prefer everything to be transparent so that, for example, JohnDoe will understand that with a higher qualification he could be earning $x more for the same assignment–not because I want to penalize JohnDoe but because if I don’t know John I have to triangulate the value of his work more.

    I agree, concept selling Mturk to one’s client/team/investor is a communication challenge. To some extent, it should be: if you can’t defend your plan in words, how are you going to successfully execute? But I think what you’re describing is their refusal to grasp your compelling algorithm and financial argument. Until the zeitgeist catches up with Mturk anything short of a brilliant business proposition is likely to be met with undo skepticism. It is important to be skeptical, but it is lazy not to think more than one or two moves in advance. If your presentation is solid, and your audience still isn’t impressed by the Mechanical Turk, I question their chess skills.

  4. labsji says:

    Hey Ryan,
    @How to make payments bypassing the mturk 10% cut:
    With little effort you can do that in main grid itself.
    Have you seen, it takes in your AWS requester account identity and sucks in all transaction related to your requester account. A similar, can be built.
    Build your HIT such that it takes email id also so that you can make paypal payments. The hit will also specify that payment is by ‘’. Technically, pass info to the service between xml tags.Make it a 1 cent hit so that amazon has reasons to keep the HIT alive. will constantly poll for requests and results and make paypal payment automatically(upon HIT approval) to the email account specified. The payment history and result quality assessment can be encoded as Qualification and posted in Mturk.

    @Concept selling:
    The idea I was mooting with limited success is to be
    a reputed Mtruk intermediary( with an Indian slant). Just like intermediaries that add value in other AWS services are thriving, there is lot of scope for Mturk intermediary. For example there are many online backup providers that add value on top of S3.
    Designing the HIT, managing the request, approval process is complex enough. Thus there is enough justification for a Mturk intermediary. For an objective evaluator, the Mturk activity levels are not compelling enough to jump in right away.
    As you mentioned, Zeitgeist is what will add momentum, And that is what I’m working on.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s