Privacy chat 2009-09-29

(10:50:44 PM) evanpro: Hey, so, privacy
(10:50:49 PM) evanpro: Is haaaaaaaaaaaaaaard
(10:50:55 PM) evanpro: I know it seems not hard
(10:50:59 PM) evanpro: It's _hard_
(10:51:00 PM) candrews: Yar!
(10:51:11 PM) candrews: well, it wouldn't be hard if we didn't have to worry about caching.
(10:51:17 PM) evanpro: It's still hard
(10:51:45 PM) evanpro: I think I mentioned talking to Twitter folks about it and they said, "If you can at all avoid it, just don't do it."
(10:51:54 PM) candrews: I do recall you saying that.
(10:51:57 PM) evanpro: B-)
(10:51:57 PM) funkatron left the room (quit: ).
(10:52:07 PM) candrews: That poor contributor picked the wrong thing to start on!
(10:52:14 PM) evanpro: Yeah, I'm sorry to say it
(10:52:53 PM) candrews: Privacy is probably so hard, it's the kind of problem that's going to require a few people, a white board, and probably a POC or 2 to get right.
(10:53:01 PM) candrews: is it a 0.9 requirement?
(10:54:58 PM) evanpro: It's on the roadmap
(10:55:13 PM) evanpro: It's fallen off the requirement list for 2-3 releases... at least since 0.6
(10:55:50 PM) evanpro: so, let me see if I can kind of sketch out how I think it could be done _wrong_
(10:56:00 PM) evanpro: This is how I believe Twitter did it
(10:56:16 PM) evanpro: So for every stream you're looking at... say, http://identi.ca/tag/privacy ...
(10:56:43 PM) evanpro: You query for a pageful of notices (21)...
(10:56:47 PM) evanpro: ...and for each notice...
(10:56:55 PM) evanpro: ...you determine if the current user has any right to see it.
(10:57:45 PM) evanpro: Namely, if the author has marked their stream "private", you determine if the current user "follows" the author, and if so, then you show the notice.
(10:57:51 PM) candrews: that doesn't work, cause then you can end up with pages that show varying numbers of notices, which will look really weird.
(10:58:09 PM) evanpro: If you have less than 20 notices at the end, you get another slice and sift through them again.
(10:58:37 PM) candrews: Which gets very expensive as you get to higher number pages (the notices to display on page 5 will be a lot of work to compute)
(10:58:45 PM) evanpro: Right
(10:59:07 PM) candrews: You could write a database query to do; it's a series of joins
(10:59:17 PM) evanpro: You're also doing at least a few extra expensive queries for each notice
(10:59:39 PM) evanpro: "it's a series of joins" => "it will murderate your database"
(11:00:01 PM) evanpro: results can't be cached, of course, because each user is going to see a different stream
(11:00:13 PM) candrews: yep... i bet you'd need quite the database server cluster to handle the volume of traffic identica gets in that case.
(11:00:57 PM) candrews: So sql joins don't really work, and doing the privacy determination at render time sucks too... where does that leave us?
(11:01:07 PM) evanpro: Finally, you'd have to send "private" flags across with the OMB subscription information
(11:01:40 PM) evanpro: And when it changes, you'd have to push that across with the update profile notification.
(11:02:14 PM) evanpro: Assuming (!) that the subscribing server respects the privacy flags, and doesn't share out private notices accidentally or on purpose.
(11:02:33 PM) candrews: and that change, I assume, would retroactive apply to all previous notices by that author. Actually, that's a really interesting scenario.
(11:02:50 PM) evanpro: So, let's talk about what we can do to make this a little less crazy.
(11:03:33 PM) evanpro: 1) let's make the reasonable extrapolation that _privateness_ applies to a notice
(11:04:02 PM) evanpro: So we either add a new field or overload the already-groaning is_public flag with another value.
(11:04:41 PM) candrews: I'm with you. And like notice content, privateness of a notice is immutable.
(11:04:53 PM) evanpro: 2) Let's accept the very reasonable restriction that private notices are only visible in the inbox, in the replies tab, in the profile page, and in favorites
(11:05:02 PM) evanpro: candrews: yes, immutable!
(11:05:29 PM) evanpro: That is, everyone who looks at the public timeline sees _only_ non-private notices
(11:06:30 PM) evanpro: Also, if I look at your inbox (which I can!), and hellekin has posted a private notice, and both of us are subscribed to hellekin, I still won't see the notice there!
(11:07:02 PM) evanpro: Then things get a little more tractable
(11:07:14 PM) candrews: why is that? What does that buy us?
(11:07:29 PM) evanpro: Well, then, we only have to have two different versions of /candrews/all
(11:07:35 PM) evanpro: Your version, and everyone else's
(11:07:36 PM) shiny: how aobut we grab up tp 10 public, and up to 10 private, then order them our sel ves
(11:07:57 PM) evanpro: candrews: much more cache-friendly
(11:08:36 PM) shiny: our orderby is by notice id, right?
(11:08:55 PM) evanpro: we'll have to be more careful in our queries for public stuff (search, public timeline, popular, tag, group, etc.) to filter out stuff that is private
(11:09:10 PM) evanpro: but I think that's tractable
(11:09:28 PM) shiny: the "pull out 20 show 10" seems to work for me
(11:09:29 PM) evanpro: Anyways, if we can accept those restrictions, then we can actually handle privacy pretty nicely
(11:09:54 PM) candrews: I wonder if the /user/all optimization is going to be confusing for the average user.
(11:10:22 PM) evanpro: I'm not sure
(11:10:44 PM) evanpro: considering that most Twitter users are unable to view any page besides their own inbox, probably not
(11:10:45 PM) candrews: hmm, how would group pages work? if you post a private notice to a group, and I'm authorized to see it, do I see the notice on the group page, or not?
(11:11:07 PM) evanpro: I'd say no
(11:11:22 PM) candrews: the more I hear about this twitter thing, the more I amazed I get that people actually use it. It seems so feature-poor and horribly designed :-)
(11:11:33 PM) evanpro: But that gets into a very trickful issue
(11:11:53 PM) evanpro: Which is that one of the most-requested features I get from companies is private groups
(11:12:03 PM) evanpro: That is, a group for which all notices are private
(11:12:18 PM) shiny: meaning, if !groupname is in there, then it's private?
(11:12:22 PM) shiny: a per notice requirement?
(11:12:27 PM) evanpro: Right
(11:12:33 PM) shiny: have you used yammer much?
(11:12:47 PM) shiny: a notice into a group is a different creature to a notice to your timeline
(11:12:58 PM) shiny: none of the !groupnamr syntax either
(11:13:14 PM) ***shiny is converting a telco from yammer to statusnet this week
(11:13:15 PM) evanpro: So "!executives let's lay everyone off" can only be read by people in the group
(11:13:19 PM) evanpro: !!!!!
(11:13:25 PM) evanpro: shiny: really!?
(11:13:32 PM) evanpro: shiny++
(11:13:42 PM) shiny: yea, they hate the yammer desktop client
(11:13:47 PM) shiny: nothing to do with freedom
(11:13:48 PM) candrews: wow, if someone forgets the "!" or tries sending a message to a group they forgot to join, that could be incredibly awkward :-)
(11:13:50 PM) evanpro: What client will they use?
(11:13:56 PM) evanpro: candrews: yeah
(11:14:08 PM) shiny: evanpro: firefox one is popular
(11:14:11 PM) evanpro: Well, people send email to the wrong recipients all the time