Categories
Projects Speaking

PixelPhones – a huge display made with smartphones

Whew!

It’s been a massively intense few weeks getting my PixelPhones project ready in time for my presentation at FOTB. I’ve been coding day and night for months, and I was up until 4am the night before making sure that I’d thought of everything.

The worst part was that I had no idea if it would work – it’s kinda difficult to get hundreds of phones together to test it. But it did work – and the results were better than I could have hoped for!

thanks to Momo for the amazing video

PixelPhones is this crazy project where I turn each phone in the audience into pixels in a large display. It runs in your phone browser so you don’t need to install an app.

Built with openFrameworks in C++, the app sets up its own WebSocket server and phones connect straight into it over the local wifi network (although in future I hope to get this working over 3G).

Of course socket connections are two way so I couldn’t help but take this chance to involve the audience too – I made a game where Nyan Cat runs from screen to screen, and whoever catches him fastest wins! (partly inspired by Rob Davis’ Cat on Yer Head game)

The tricky bit was figuring out where the phones are – GPS just isn’t accurate enough. I was inspired by the Junkyard Jumbotron – it puts markers on each phone, then you take a picture of them all. But in my app a marker would be way too small so instead I use a unique sequence of flashes.

It uses OpenCV to find the flashing objects, read the pixels, and decode the sequence.

Latency was also an issue, even on the fastest networks the packets arrive at slightly different times. And when you want super fast smooth animation, unpredictable timing is unacceptable. I solved it thanks to Jobe Makar’s excellent method of syncronisation outlined in his ActionScript multi-player gaming book.

I’m so happy and relieved that it worked so well and the timing was split-second perfect. The next day I put myself through the whole ordeal again, this time with a larger audience and only 10 minutes to demo (at the FOTB Jam session). Over 220 devices connected and I could have handled more but we ran out of time.

This is just the start, and I have many many ideas that I have yet to try. I’ll also be open sourcing this code when it’s finished – I can’t wait to see what the rest of you do with it! Both sessions were recorded in full, I’ll let you know when they’re uploaded.

40 replies on “PixelPhones – a huge display made with smartphones”

That really is very cool.
I would like to see a video of the Nyan cat game. Wish I could of been there!

Thanks Phil! The game is just literally the Nyan Cat gif moving across the screen, so not that much to see. 🙂 If you look in Momo’s video you can see at the end the screens lighting up as the cat runs past!

After hearing about this on the Creative Coding Podcast, it’s awesome to see the idea in action – incredible!

I heard last week that you were doing this based on the Junkyard Jumbotron, so I actually started working on my own and knowing that it would be way too difficult to read hundreds or thousands of QR Codes I decided to go down a path of using a 5-character string flashed in morse code. Now that you’ve posted this I see you’ve done much the same. Did you add color to your flashing code, and if so do you think that there could be some issues when brought into an environment that isn’t as dark as an auditorium? Also I decided to do mine in flash with a Red5 server, mainly because I couldn’t deal with the browser window from peoples phones/tablets.

Hi Ryan,

I didn’t use colour, no – it’s too unpredictable and hard to read. I’ve tested it in natural daylight and it seems to work OK. Biggest problem is reflected light. And it’s really important to me to get it running in a browser – I reckon I can get rid of the browser bar with a bit of work, or at least if they install it as a web app. I just ran out of time.

Seb

Hi Mike, yep I am in fact doing that but as it was such a last minute rush I forgot to make the page big enough that there was something to scroll. It’s fixed now. 🙂 Also I’m adding a pop-up to ask people to add it to their home screen – that way I can make the status bar black – even better!

Hi Seb,

Interestingly, we’ve done a very similar project at Carnegie Mellon (though we didn’t get a chance to test it with 100 people sadly). I am wondering how similar our system is to yours. Our system is also web based (you just go to a webpage to flash lights and be a part of the display), and we use a camera. We identify pixels using unique flashing colors (rgb) which encodes a unique id for every phone. Each phone flashes this sequence until it is identified and assigned a color (or sequence of colors for animation). Is this how your system works?

Thank you for clarifying.

Hi Julia! It does seem quite similar – I’d love to see how yours works! Do you have any video of it online? I guess with projects like Junkyard Jumbotron it was only a matter of time before people did this on a larger scale.

My ID broadcast system just uses black and white flashes and was designed to work in large venues where a single phone could only be a few pixels big. And in theory works with 1000s of different IDs.

I assumed that it wouldn’t be possible to accurately read colour data from the phones from that distance, and it certainly seems that colours can get burnt out very easily.

Also the phones are connected in real-time using WebSockets so I can create interactive animations and patterns.

I’m going to be at Art&&Code at CMU in a couple of weeks – we should get together and compare notes. Do you know Golan Levin?

cheers

Seb

Love it! However, it would be easier/quicker to show a unique 3 or 4 digit code on everyone’s phone and ask the audience to type in the code of the person’s phone to their left. Then ask them to type in the code of the phone behind them (only people in the first column of seats need to do this, but it would compensate for gaps if everyone did it).

It would take a second for the system to compute positions and maybe 30 seconds for the audience to follow the instructions. It won’t be as cool as the system “seeing” the phones but the result should be a much faster setup time and 100% accuracy to include all phones.

Thanks Gary! That would work I guess, but I wanted a system that was a bit more flexible, that would work in more ad hoc, less organised crowds. I’m working on improving the detection algorithm now, so hopefully it should be very robust! Wish me luck 🙂

Awesome work, gutted to miss flash on beach this year! always following your latest work man!
By the way is Pv3d project dropped?
Good luck with all the latest work

I’d love to hear the details on how you synchronized the animation. Are you timestamping the animations and using the local clock?

I am, yes, although the client needs to know the time relative to the server so there’s a process that works out the difference. I got the technique from Jobe Makar’s Multiplayer Games ActionScript book.

Hi. I’m not sure if you’ve noticed, but at Mobilism there where lamps that flashed from time to time and your system attempted to find them.

Maybe you could get a faster localization of phones if you would flash all of them with the same code (or random code with some seed) at first and automatically change a poly/region in which the phones were recognized to be “yours”.

BTW. Sadly on Firefox mobile on Nokia N9 (MeeGo) the app didn’t work (I had a black screen with some port number).

Hi Nux,

flashing lights aren’t really a problem to be honest, as they’re quickly discounted by the system. It automatically finds anything flashing so there’d be no benefit from having an additional code broadcast at the start.

Sorry to hear about the N9 – in fact I’ve never used the browser on that – I’m guessing it doesn’t support WebSockets. More investigations required!

thanks for the comments,

Seb

Do all the devices need to be using the same wi-fi access point? I’m wondering about use in a large theater that could hold 6,000.

Hi Kevin,

They don’t have to be on the same access point, but it helps if they’re on the same network. WiFi is always the hardest part with this project, but if you can provide reliable wifi for 6000 people then it should work! Perhaps I could help? Give me a mail seb at seb.ly.

Cheers!

Seb

Comments are closed.