Privacy chat 2009-09-29

(10:50:44 PM) evanpro: Hey, so, privacy (10:50:49 PM) evanpro: Is haaaaaaaaaaaaaaard (10:50:55 PM) evanpro: I know it seems not hard (10:50:59 PM) evanpro: It's _hard_ (10:51:00 PM) candrews: Yar! (10:51:11 PM) candrews: well, it wouldn't be hard if we didn't have to worry about caching. (10:51:17 PM) evanpro: It's still hard (10:51:45 PM) evanpro: I think I mentioned talking to Twitter folks about it and they said, "If you can at all avoid it, just don't do it." (10:51:54 PM) candrews: I do recall you saying that. (10:51:57 PM) evanpro: B-) (10:51:57 PM) funkatron left the room (quit: ). (10:52:07 PM) candrews: That poor contributor picked the wrong thing to start on! (10:52:14 PM) evanpro: Yeah, I'm sorry to say it (10:52:53 PM) candrews: Privacy is probably so hard, it's the kind of problem that's going to require a few people, a white board, and probably a POC or 2 to get right. (10:53:01 PM) candrews: is it a 0.9 requirement? (10:54:58 PM) evanpro: It's on the roadmap (10:55:13 PM) evanpro: It's fallen off the requirement list for 2-3 releases... at least since 0.6 (10:55:50 PM) evanpro: so, let me see if I can kind of sketch out how I think it could be done _wrong_ (10:56:00 PM) evanpro: This is how I believe Twitter did it (10:56:16 PM) evanpro: So for every stream you're looking at... say, http://identi.ca/tag/privacy ... (10:56:43 PM) evanpro: You query for a pageful of notices (21)... (10:56:47 PM) evanpro: ...and for each notice... (10:56:55 PM) evanpro: ...you determine if the current user has any right to see it. (10:57:45 PM) evanpro: Namely, if the author has marked their stream "private", you determine if the current user "follows" the author, and if so, then you show the notice. (10:57:51 PM) candrews: that doesn't work, cause then you can end up with pages that show varying numbers of notices, which will look really weird. (10:58:09 PM) evanpro: If you have less than 20 notices at the end, you get another slice and sift through them again. (10:58:37 PM) candrews: Which gets very expensive as you get to higher number pages (the notices to display on page 5 will be a lot of work to compute) (10:58:45 PM) evanpro: Right (10:59:07 PM) candrews: You could write a database query to do; it's a series of joins (10:59:17 PM) evanpro: You're also doing at least a few extra expensive queries for each notice (10:59:39 PM) evanpro: "it's a series of joins" => "it will murderate your database" (11:00:01 PM) evanpro: results can't be cached, of course, because each user is going to see a different stream (11:00:13 PM) candrews: yep... i bet you'd need quite the database server cluster to handle the volume of traffic identica gets in that case. (11:00:57 PM) candrews: So sql joins don't really work, and doing the privacy determination at render time sucks too... where does that leave us? (11:01:07 PM) evanpro: Finally, you'd have to send "private" flags across with the OMB subscription information (11:01:40 PM) evanpro: And when it changes, you'd have to push that across with the update profile notification. (11:02:14 PM) evanpro: Assuming (!) that the subscribing server respects the privacy flags, and doesn't share out private notices accidentally or on purpose. (11:02:33 PM) candrews: and that change, I assume, would retroactive apply to all previous notices by that author. Actually, that's a really interesting scenario. (11:02:50 PM) evanpro: So, let's talk about what we can do to make this a little less crazy. (11:03:33 PM) evanpro: 1) let's make the reasonable extrapolation that _privateness_ applies to a notice (11:04:02 PM) evanpro: So we either add a new field or overload the already-groaning is_public flag with another value. (11:04:41 PM) candrews: I'm with you. And like notice content, privateness of a notice is immutable. (11:04:53 PM) evanpro: 2) Let's accept the very reasonable restriction that private notices are only visible in the inbox, in the replies tab, in the profile page, and in favorites (11:05:02 PM) evanpro: candrews: yes, immutable! (11:05:29 PM) evanpro: That is, everyone who looks at the public timeline sees _only_ non-private notices (11:06:30 PM) evanpro: Also, if I look at your inbox (which I can!), and hellekin has posted a private notice, and both of us are subscribed to hellekin, I still won't see the notice there! (11:07:02 PM) evanpro: Then things get a little more tractable (11:07:14 PM) candrews: why is that? What does that buy us? (11:07:29 PM) evanpro: Well, then, we only have to have two different versions of /candrews/all (11:07:35 PM) evanpro: Your version, and everyone else's (11:07:36 PM) shiny: how aobut we grab up tp 10 public, and up to 10 private, then order them our sel ves (11:07:57 PM) evanpro: candrews: much more cache-friendly (11:08:36 PM) shiny: our orderby is by notice id, right? (11:08:55 PM) evanpro: we'll have to be more careful in our queries for public stuff (search, public timeline, popular, tag, group, etc.) to filter out stuff that is private (11:09:10 PM) evanpro: but I think that's tractable (11:09:28 PM) shiny: the "pull out 20 show 10" seems to work for me (11:09:29 PM) evanpro: Anyways, if we can accept those restrictions, then we can actually handle privacy pretty nicely (11:09:54 PM) candrews: I wonder if the /user/all optimization is going to be confusing for the average user. (11:10:22 PM) evanpro: I'm not sure (11:10:44 PM) evanpro: considering that most Twitter users are unable to view any page besides their own inbox, probably not (11:10:45 PM) candrews: hmm, how would group pages work? if you post a private notice to a group, and I'm authorized to see it, do I see the notice on the group page, or not? (11:11:07 PM) evanpro: I'd say no (11:11:22 PM) candrews: the more I hear about this twitter thing, the more I amazed I get that people actually use it. It seems so feature-poor and horribly designed :-) (11:11:33 PM) evanpro: But that gets into a very trickful issue (11:11:53 PM) evanpro: Which is that one of the most-requested features I get from companies is private groups (11:12:03 PM) evanpro: That is, a group for which all notices are private (11:12:18 PM) shiny: meaning, if !groupname is in there, then it's private? (11:12:22 PM) shiny: a per notice requirement? (11:12:27 PM) evanpro: Right (11:12:33 PM) shiny: have you used yammer much? (11:12:47 PM) shiny: a notice into a group is a different creature to a notice to your timeline (11:12:58 PM) shiny: none of the !groupnamr syntax either (11:13:14 PM) ***shiny is converting a telco from yammer to statusnet this week (11:13:15 PM) evanpro: So "!executives let's lay everyone off" can only be read by people in the group (11:13:19 PM) evanpro: !!!!! (11:13:25 PM) evanpro: shiny: really!? (11:13:32 PM) evanpro: shiny++ (11:13:42 PM) shiny: yea, they hate the yammer desktop client (11:13:47 PM) shiny: nothing to do with freedom (11:13:48 PM) candrews: wow, if someone forgets the "!" or tries sending a message to a group they forgot to join, that could be incredibly awkward :-) (11:13:50 PM) evanpro: What client will they use? (11:13:56 PM) evanpro: candrews: yeah (11:14:08 PM) shiny: evanpro: firefox one is popular (11:14:11 PM) evanpro: Well, people send email to the wrong recipients all the time