skud: (skud)

Mirrored from Infotropism. You can comment there or here.

A few times on the Growstuff mailing list or IRC channel, someone’s excitedly suggested that we should import data from another CC-licensed data set. Each time, I say, “Trust me, that’s pretty complicated,” but I’ve never actually sat down and explained the full gory details of why.

The following is something I wrote up for our wiki so that I could point people at it next time the subject comes up. I thought it might be interesting to a wider audience, too, so that’s why I’m posting it here.

Importing data is hard.

This is a bit of a rant by Skud, who used to work on Freebase, a large open-licensed data repository which imported data in bulk from a range of sources, including Wikipedia, Netflix, the Open Library Project, and many more. She’s had a lot of experience in this area, and learnt a lot about the weird complications of mass data imports.

The simple case

They have a database. You have a database. Your fields are the same. Their API is easy to use and their license is compatible.

  1. map their fields to ours, eg. their “name” is our “system_name”
  2. import data
  3. PROFIT!

Their fields aren’t the same

What if the fields aren’t quite equivalent? For instance, let’s say they have measurements in imperial and we use metric. We’ll need to have ways to convert them. That’s actually a really simple example. Import incompatibilities are more often at a semantic/ontological level. Growstuff has the idea of “crops” and “varieties” but what if the other database only has “plants” with no distinction? Or what if they have crops and varieties but draw the line somewhere slightly different to where we do? These sorts of incompatibilities are more common than not, and massively complicate any import effort.

Some of their data is bogus

Nothing against that other database — some of everyone’s data is bogus! But we need to check it. What “bogus” means will vary from place to place, but it might be spam entries, duplicate records, simple errors, or it might be cruft from their own broken imports. We need to look carefully at every import and make sure we’re skipping as much of this as possible. And this is largely a manual process, since what the bogosity will never be the same twice. You can do this by sampling, of course, but you still need to look at something on the order of a hundreds of records, and know what you’re looking for. Could you spot a mixed-up scientific name on a randomly chosen herb? I couldn’t.

The stub problem

Let’s say we want to import from a database of plant life that lists 10,000 edible plants and their nutritional content. Growstuff has 300 crops at present. We import everything! Now we have 9,700 pages with nothing but nutritional data. Nobody on Growstuff is using them, they have no pictures, they have no planting data, they have no discussions (except maybe spam comments that nobody cleans up because nobody notices). Our “newest crops” page, usually a source of interest, is now just a wasteland of grey placeholder images.

Should we have imported all 10,000 plants, or just the nutritional data of the 300 we already have? Or something in between? The answer is usually “something in between” — you might want that data if and only if you can get other partial data from other imports to make it more interesting.

The best way to do this is to import the 300 and make a note of the 9700. Then later, you can cross-correlate the notes you’ve made from various data imports and re-import those that have, say, at least 3 useful data sources and a picture. But that’s pretty complicated. (Also, see the discussion of repeated imports, below.)

Don’t forget the license

Let’s assume that their data is licensed compatibly — that means CC-BY-SA or CC-BY in our case, since we’re CC-BY-SA and none of the other clauses (ND, NC) are compatible with us. (Ignoring CC-0 and public domain stuff for now — those don’t need attribution at all.)

So by importing, we have to credit them. Now we need some way to represent that in the database. If we do this at the object level, it’s fairly simple: each thing in the database (crop, etc) has many licensors, each of which includes a name for the work (eg. “Katie’s Plants”), a license (eg. CC-BY), a licensor name (eg. Katie Smith), and a URL to link to the original data.

Now we have to display them on the page. Where? Probably at the bottom somewhere: “Some information on this page came from: Katie’s Plants (that would be a link) — CC-BY Katie Smith; SuperPlantDB under CC-BY-SA SuperPlants Inc; etc.”

The license chaining problem

Now imagine that the data on those sites came from other sites. For instance, let’s say Katie’s Plants previously did an import from Freebase.com, and SuperPlantDB did one from Wikipedia. We not only need to credit Katie’s Plants and SuperPlantDB but also those places.

Some questions to consider:

  • Are those second-degree sources, their licenses, licensors, etc available via the API? When Freebase imported images from Wikimedia Commons, we encountered this problem, because the license metadata had to be scraped from inconsistently-formatted HTML. Getting this wrong leads to complaints from licensors whose licenses we’re violating.
  • Do we know what part of the data on Katie’s Plants was sourced from Freebase? Maybe it was the international names, but we’re importing medicinal uses and not touching that part of her data. Does Katie’s license notice express this? Probably not — there’s no requirement in the CC licenses for the attribution to be at the field level, and our own attribution notices definitely don’t operate at this level of detail. Because we don’t have the detail, this means we end up with attribution inflation: pretty soon, every page on Growstuff has a hundred attributions at the bottom of every page.

Sure, we could just choose not to chain licenses, or to do it in some restricted way… but the moral high road here is to respect everyone’s license and attribution, and besides, if you only attribute some contributors, where do you decide to draw the line?

The infectious NC clause problem

This is a subset of license chaining problems. Let’s say Growstuff (a commercial entity using CC-BY-SA) imports from Katie’s Plants (a non-profit entity using CC-BY-SA) which imports in turn imports from Hippie Herbs (a non-profit entity using CC-BY-NC — note the “non-commercial” clause).

Katie’s fine — she imports from Hippie Herbs’ data with impunity because she’s non-profit. She attributes them on her site, and Hippie Herbs is happy. She doesn’t have to use the same license as them because they don’t have a “SA” (Share Alike) clause.

Now Growstuff comes along and wants to import data from Katie’s Plants. Katie’s Plants is CC-BY which is compatible with Growstuff… but what about the data that originally came from Hippie Herbs? We’re commercial, so we’re not meant to use it.

But how do we tell what’s what? Katie probably doesn’t attribute HH at the level of individual bits of data, so we can’t extract just the ok-for-commercial-use bits.

Basically, if you believe in license chaining (and as I said, it’s definitely the moral high road to take, so I think we should) then you have to be constantly vigilant for the taint of NC-licensed data anywhere in the sprawling tree of ancestors to your data.

What if we already have some data? (the merge case)

The simple case is fine for a green-field import with no existing data, which is described above. But let’s say we’re importing data into an area where we already have some contributions from Growstuff members.

  1. map the fields as before
  2. for each piece of data imported, compare with Growstuff
    1. is Growstuff’s field empty? IMPORT!
    2. are the two the same? no-op!
    3. do they differ?
  3. If they differ, do we trust our own community or the import source? Or do we need to adjudicate?

Let’s say we decide to adjudicate. We now need to build an app to let people vote on which one is “correct” — probably best of three or something like that. Freebase did this (multiple times) and I was involved in some of it. We called them “data games” and had leaderboards for who’d voted the most. We couldn’t get enough throughput, though, and sometimes by the time something had been adjudicated, another community member had edited the field on our site, thus invalidating the whole thing. We ended up paying people in developing countries to churn through these votes for us (we used ODesk, but you could use Amazon’s Mechanical Turk or whatever). However, they needed training, and weren’t cheap — even after all the work of setting up the voting queue, there was still considerable expense.

Do we let people edit the data after import?

This came up quite often with Freebase because sometimes they would import from “authoritative sources” who licensed their work specially to Freebase but didn’t generally have a CC license or an open community editing process. For instance, the time when I was talking to some people from the BBC, and one (an older dude) said, “If we gave you our programme data, we wouldn’t want anyone to edit it because we are the experts on our programmes.” This was pretty silly of course — another, younger BBC dude immediately turned to him and said “Ha ha ha, I’ve got two words for you: Doctor Who.” — but sadly these situations are common when you’re dealing with closed/non-community-based/”authoritative” data sources who don’t understand the power of crowdsourcing information.

But even when dealing with compatibly CC-licensed sources with open developer communities, there can still be some problems around the “authority” of the data and how it’s attributed.

Take the case where Katie’s Plants community have spent heaps of time editing their data and are very proud of it. We import it to Growstuff, then our community looks at it and decides that bits of it are wrong and change it.

Do we leave the license link to Katie’s Plants intact? Most likely yes, because our data has theirs in its DNA, so to speak. But what if we essentially deleted all the data from there? This might happen if, for example, we’d imported a picture from Wikimedia Commons then found that the picture was incorrect or inappropriate, so we blew it away. Now we should probably remove the license note. But how do you tell when data has been completely removed as opposed to modified or built upon?

In the Katie’s Plants example, what if Katie’s high quality medicinal plant information gets mixed up with ($DEITY forbid!) low-quality data from less experienced Growstuff members or from yet another import? What implications does this have for Katie’s site and their reputation? Under the license we’re allowed to mess it up because there is no “No Derivatives” (ND) clause, but socially/culturally they’ll be pretty unhappy if we do, and we can expect some backlash.

Repeat imports

Great news! Katie got a government grant and some fantastic press coverage, and her database has expanded enormously. We want to re-run the import. But now consider this case:

  • Katie’s plants, original: “Tomato – red”
  • Growstuff, original: “Tomato – red, yellow, green, black”

When we first imported, we put it to adjudication and found that Growstuff’s data was better, so we went with that.

Now we re-import, and Katie’s data has changed:

  • Katie’s plants, changed: “Tomato – red, yellow, green, striped”

So of course we put it through adjudication again. The correct answer is probably a union of the two sets.

Now, Katie’s database is growing fast, and so is Growstuff. We want to do a regular import from there — perhaps monthly. But somehow along the way, we’ve ended up with different ideas of tomato colour. Every month, their data is different to ours, and we have to keep re-adjudicating the same question: what colour/s are tomatoes? Boring. Our community is tired of playing the voting game, and/or it’s costing us money with our Mechanical Turk people.

So we decide to implement a check: if nothing’s changed on either side since the last adjudication, leave it. But now we have to implement change tracking, not just on Growstuff, but on Katie’s Plants as well. We need to keep a history of changes for every site we import. This is in addition to the infrastructure we’ve had to build to automatically run imports at regular time intervals.

How do we make our data available in return?

Obviously we have an API for people to access our data under CC-BY-SA. But keep in mind the license-chaining effect: if anyone uses data from Growstuff, they will also be constrained by the licenses of all the data sources we import. We will need to make that license information available in the API alongside our data, and make sure all our API docs and related materials explain the necessity of license chaining.

Take a look at Freebase’s Attribution Policy. They use CC-BY, but because of attribution chaining, they can’t just say that — they need a whole page with a wizard to help people figure out how to attribute something on the site. It’s incomplete, too: Freebase decided that they would only require license chaining for “content” as opposed to “facts” (a complicated issue in itself) which means images Wikipedia-based descriptions. They don’t require chained license information for other data sources. This is dubious in terms of the legality and culture of how Creative Commons works — there’s no really firm guidelines on this, but in my opinion the most moral/ethical stance is to always chain your attributions, and Freebase has chosen otherwise. In the past, this has caused some concern from the owners of other data sources that were imported to Freebase. Even Wikipedians have complained that Freebase doesn’t enforce their Wikipedia attributions strongly enough. This sort of thing can lead to reputation problems, if not legal ones.

Just the facts, ma’am

One final complication. Various courts have ruled that “facts” aren’t copyrightable. For instance, the fact that the crop “Corn” has the scientific name “Zea mays” can’t be copyrighted. Even if you have thousands of these facts all together, they can’t be copyrighted, because they’re not a “creative work”. They’re just a statement of fact.

This actually throws the whole idea of CC-licensing collections of data into doubt. And yet we have nothing better, so we do it anyway.

Some data projects have come up with various justifications for this. For instance, Freebase says that the arrangement of the facts is a creative work — that what’s CC-licensed is their schema. That’s pretty creative in itself! The thing is, none of this has really been tested. And so most open data projects have some kind of Terms of Service which explains what they think the CC-license is for and how it’s meant to be used. These generally say, “By accessing our data via our website or API, you agree to behave as if this CC license applied to it (even if there’s not a very strong legal basis for that outside this TOS).”

The original idea of CC licenses was to stop people having to write their own terms and conditions of use for their work, and standardise in such a way that people could easily re-use creative content. Yet for data projects, we end up having to make up our own TOS just to apply a CC license, and we’re back where we started — having to peer at a bunch of legalese and figure out what the hell it means.

Of course once you get into the complexities of license chaining described above, you now also have TOS chaining — if Growstuff uses Katie’s data under their TOS, and Katie uses Hippie Herbs’ under their TOS, is Growstuff now subject to Hippie Herbs’ TOS? No idea! I am not a lawyer! I don’t want to be one! I just want to make a website about growing food!

Conclusion

Importing data is hard! That doesn’t mean we shouldn’t do it, but we should go into it with an awareness of the potential potholes, and carefully weigh up whether importing something is the best choice for us at any given time.

Final note

Katies Plants, Hippie Herbs, and SuperPlantDB are all made-up examples. Any resemblance to actual open data projects is coincidental. Freebase, Wikimedia Commons, and the BBC are real, though.

skud: (skud)

Mirrored from Infotropism. You can comment there or here.

You think those Google recruiters would know not to contact me, but the other day I got another perky “Opportunities at Google” email from one of them, telling me that they’d found my “online profile” and that based on my experience they think I “could be a great addition to our team!”

Riiiiight.

Since I just deleted my LinkedIn profile, I emailed them asking where they’d found this “online profile”, since it was obviously outdated. Oddly enough, it seems they’d found a page about me on the Geek Feminism Wiki, and were using the rather sketchy outline of my open source background there as justification for trying to recruit me.

The recruiter admitted that the page was out of date, and asked me to let them know what I’d been up to lately so they could add it to their records. Below is a copy of what I sent them. I’m posting it here, lightly edited, for anyone who’s interested, and in the hopes that the next Google recruiter (I have no doubt that there’ll be one) might use that web search thingamajig to find out whether I’m a suitable candidate before emailing me.


Here’s what I’ve been up to for the last couple of years, since you asked.

In July 2010 the startup I was working for, Metaweb, was acquired by Google. I was brought in on a 1-year fixed term employment contract, since the group we were acquired into (Search) didn’t really know what to do with a technical community manager. I attempted to transfer my role over to Developer Relations, but was told that I “wasn’t technical enough” for the job I’d been doing for 3+ years, presumably because I didn’t have a computer science degree and believed that supporting our developer community was more important than being able to pass arbitrary technical quizzes.

Around the same time, Google started to develop Google+. As a queer/genderqueer woman, victim of abuse, and someone who was (at that very time) experiencing online harassment and bullying, I was very vocal within Google for the need for Google+ to support pseudonymity. Google decided not to do that, and instead told people they should use “the name they are known by” while in actual fact requiring their full legal names, in many cases requiring people to provide copies of their government ID when challenged. (Extensive documentation about this is available on the Geek Feminism wiki, if you’d like to read it. See Who is harmed by a “Real Names” policy? for starters.)

When I walked out the door of Google’s San Francisco office on July 15th, 2011, I was very glad to have left a company I thought was doing evil towards any number of marginalised and at-risk people. My first tweet on leaving was to criticise them for it.

Less than a week later I got my first email from a Google recruiter — not first ever, of course; I’d been spammed with them for years, but first since I quit working for them. Here’s the blog post I wrote about it. In case you can’t be bothered clicking through and reading it, here’s the money shot:

If you are a Google recruiter, and you want me to interview for SWE or SRE or any role that has an algorithm pop quiz as part of the interview, if you want me to apply for something without knowing what team I’ll be working on and whether it meshes with my values and goals and interests, if you want me to go through your quite frankly humiliating interview process just to be told that my skills and qualifications — which you could have found perfectly easily if you’d bothered to actually look before spamming me — aren’t suitable for any of the roles you have available, then just DON’T.

The very day after I blogged about that, my Google+ account was suspended, for using the name I was almost universally known by. Over the next couple of months, I campaigned tirelessly for Google+ to change its policies, working with the EFF and other advocates. My work was covered in Wired, The Atlantic, and a number of other mainstream press outlets. Obviously this was to no avail as Eric Schmidt (at the time, CEO of Google) described pseudonymous users like me as “a dog or a fake person” and no substantive change has ever been made to allow pseudonymous use of the service, despite promises to do so.

I returned to Australia and went back to school. I did a semester of Sound Production at TAFE, but it turned out that the sound engineering course I was enrolled in wasn’t really my cup of tea, just like I’d previously decided, back in the ’90s, that university wasn’t for me. Like so many others, I quit my computing degree because I was more interested in the Internet and open source software than in fixing COBOL applications for banks who were worried about Y2K. But then, I’m sure Google’s HR system already knows all about that — if I’d had a degree, you might have considered me worth keeping on last year. Instead, Google’s reliance on higher education credentials causes it to weed out people like me, even though I have a track record a mile long and buckets of evidence to show that I’m good at what I do.

In the end, I’ve spent most of the last year lying in hammocks reading books, working in my garden, going to gigs, hanging around recording studios, doing the odd bit of freelancing, and, over the last few months, travelling around Europe. It’s given me a good opportunity to reflect on my previous work.

Since I’ve been out of the Silicon-Valley-centred tech industry, I’ve become increasingly convinced that it’s morally bankrupt and essentially toxic to our society. Companies like Google and Facebook — in common with most public companies — have interests that are frequently in conflict with the wellbeing of — I was going to say their customers or their users, but I’ll say “people” in general, since it’s wider than that. People who use their systems directly, people who don’t — we’re all affected by it, and although some of the outcomes are positive a disturbingly high number of them are negative: the erosion of privacy, of consumer rights, of the public domain and fair use, of meaningful connections between people and a sense of true community, of beauty and care taken in craftsmanship, of our very physical wellbeing. No amount of employee benefits or underfunded Google.org projects can counteract that.

Over time, I’ve come to consider that this situation is irremediable, given our current capitalist system and all its inequalities. To fix it, we’re going to need to work on social justice and rethinking how we live and work and relate to each other. Geek toys like self-driving cars and augmented reality sunglasses won’t fix it. Social networks designed to identify you to corporations so they can sell you more stuff won’t fix it. Better ad targetting or content matching algorithms definitely won’t fix it. Nothing Google is doing will fix it, and in fact unless Google does a sharp about-turn, they’ll only worsen the inequality and injustice there is in the world.

I guess you’ll want to know what I’m working on at the moment. My current project is an open source, open data project called Growstuff, which helps food gardeners track and share information about what they’re growing and harvesting. It is built on principles of sustainability, including a commitment to a diverse and harassment-free community, to actively supporting developers rather than excluding them based on misguided ideas of meritocracy, and to funding the project through means that will never put the people running the website in opposition to our customers. That means no ads, in case you’re wondering. We’d rather our members paid us directly; that way, we’ll never forget who we’re meant to be serving. I’m working on Growstuff from home, where I can be myself and feel safe and comfortable. I work with volunteers from all round the world, and get to teach programming and web development and system administration and project management and sustainability to all kinds of people, especially those who’ve previously been excluded from or marginalised in their technical education or careers. We get to work on things we know are wanted and appreciated, and we don’t have to screw anyone around to do it.

Let me know when Google has changed enough to offer me something more appealing than that. If you don’t think that’s likely to happen, then please put me on whatever “Do Not Contact” blacklist you might have handy. I know you must have some such list; I only wish you regularly referred to it instead of spamming people who not only don’t want to work for you, but have nightmares about it.

skud: (Default)

Mirrored from Infotropism. You can comment there or here.

Look, I may as well post about it. I’ve been planning it for months, and a whole swag of people already know, but this’ll make it official.

Sometime around early September, I’m planning on heading back to Melbourne, Australia, whereupon I hope to spend a few months bumming around on people’s sofas/the beach/relatives’ farms/etc, before going back to school in 2012 to study sound engineering.

Q&A time…

So I’m leaving Google, then? Yup, that’s the plan. I’ll have done a year there since Metaweb’s acquisition, and I’ve got a lovely new replacement, Shawn, who started a couple of weeks ago and who’ll be supporting the Freebase developer community going forward.

Why sound engineering? Because it gets me away from the tech industry, from sitting in an office all the time, and from the mind-boggling ennui that’s started to attack me whenever I think about software and the development thereof. It’s well past time for a change. And I’ve been enjoying myself so much volunteering at Gilman St that it seemed like something I’d like to pursue more seriously. Plus, it’s a field that’s at the intersection of technical/creative that really works for me, and I suspect that with the increasing digitisation of sound production my computing background will serve me well.

What sort of work do I want to do, then? I’m not going to commit to anything at this point, but stuff with a “startup” feel to it (to use the tech industry term), that harnesses grassroots participation and encourages disintermediation between artists and fans really appeals to me. You know the stuff I like — open culture, remix and transformative works, online collaboration and crowdsourcing, micro-entrepreneurism, activism, connecting people together. If I can’t find a way to mix that stuff with a background in Internet technologies and a fresh education in the tech side of music production, I’ll be very surprised.

Why Australia? Why not go to school in the US? Short answer: tuition in Australia is about 5% of what it is in the US for similar sorts of courses, and I won’t need a visa for it.

What school? What program/course? I’m looking at a Certificate IV and Advanced Diploma in Sound Production, which is a 2 year course offered by various TAFEs (Technical and Further Education institutions — UK readers please think “Polytechnic”, US readers please think of a cross between a community college and DeVry). RMIT’s course description gives a pretty good overview of the program. I’m also considering NMIT. If anyone happens to know anything about those two institutions/courses and can offer advice or opinions, they’d be very much appreciated. (Yes, I’ve emailed the faculty/admissions for both; no, I can’t make it to Open Day at either.)

Will I be doing X before I leave? (For values of X usually including certain conferences or places to visit.) I’m attending WisCon in Madison, WI in just over a week, and will probably be in Portland in late July during OSCON though not attending (I do hope to catch up with a bunch of my friends there, though). I am not planning to attend any other conferences/events between now and when I leave, nor do I have plans, or much time, for other travel at this point.

Will I be coming back to the US after completing my study? Maybe. The sort of work I want to do (see above) may lead me back to the Bay Area, if the visa-granting gods smile on me. Who knows? It’s also likely that even if I don’t move back here, I will visit occasionally if my budget allows.

And this is definitely definite? Well, it’s about 90% definite at this point. It’s possible that something might happen to completely change my mind in the next couple of months, but I wouldn’t hold my breath for it.

So here’s where I ask you for stuff.

  • If you read this and thought, “ooh, that reminds me of $person who works in that field” or “I know a startup that’s doing stuff like that” or “I bet Skud would love to hear about $project”, I would love an introduction. That goes double for anyone/anything in Australia.
  • The courses I’m applying for are quite competitive and have an application/interview process where they want to know about your previous experience in the field. So I’m interested in picking up any related work I can between now and the end of the year. Do you know anyone who needs a hand or wouldn’t mind me tagging along while they work live shows, record demos, go into the studio, or whatever? Any kind of live or recorded sound work would be of interest. Volunteer/unpaid would be preferred for now — I can’t do paid work in the US outside of my primary employment, though of course I wouldn’t turn down paying gigs once I’m back in Australia.
  • Know anyone who’s looking for a housemate in Melbourne later this year? I’m thinking of splitting a 3br house in Melbourne’s inner north (Preston?) with one other person, but I’m open to other suggestions too. Looking for a grownup who pays their bills on time, but who’s also fun to hang out with. I keep odd hours and am a bit strange, but I’m pretty considerate and reliable as a housemate, as well as being a good cook.
  • Got a spare room or need a housesitter between September and, say, Decemberish? Mostly thinking Melbourne here, and more “need someone to feed the cats for 2 weeks” than “you can crash on my sofa for a night or two”, but any and all offers would be welcome.

Please feel free to email me (skud@infotrope.net) if you can help me out with any of the above!

Mural showing a car driving on a highway, about to pick up a hitchhiker carrying a guitar.

Mural, on Treat St near 24th, in San Francisco's Mission District.

August 2014

S M T W T F S
     12
34 56789
1011 121314 15 16
17181920212223
24252627282930
31      

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags