I had the chance to test it out during the Oscars [1], receiving about 1.2M tweets via Twitter Streaming API in 6 hours, publishing them in Pub/Sub, pulling with another process and storing in BigQuery. I still need to find time to blog about the experience, happy to collect a few extra questions here.
Did you get any feel for the latency between publishing and consumption? I'm definitely interested, but I'd be looking for no more than a few second delay between publish and potential consumption.
We didn't collect precise timing data, but I hope to release the code soon, and it's pretty easy to setup a test similar to ours, e.g. with high volume hashtags such as #nowplaying.
What I can tell you is that we published messages in batches, closing a batch every 5 seconds or 100 tweets (at the highest traffic rate, we were receiving and publishing about 100 tweets every 1-2 seconds). This is way below the quota limits.
On the subscriber side, we received batches of at most 10 messages. I didn't investigate if this is a limit or a config param. The sensation was to receive the messages pretty much instantly (I was looking in parallel at 2 shells, one publishing and one receiving), but again, I don't have precise measurements.
I haven't played with the cloud version yet, but internally it's rock solid in terms of its quality. It's very highly available and reliable, extremely and effortlessly scalable, and has dependably low latency.
That's great! Looking forward to see it rolled forward to the new API revision (v1beta2, which we don't expect to change significantly). Also, please check our gcloud-node repository out, which also offers idiomatic Cloud Pub/Sub library and the team is planning to move to v1beta2.
I’d suggest Firebase for scenarios that need real-time notification all way to a device, app, or browser.
Google Cloud Pub/Sub is better aligned to server-to-server messaging, similar to other Cloud queuing services, service buses, event logs, or open-source systems such as RabbitMQ or Kafka.
During the beta period, the service is available for free. Once it comes out of beta, developers will have to pay $0.40 per million for the first 100 million API calls each month. Users who need to send more messages will pay $0.25 per million for the next 2.4 billion operations (that’s about 1,000 messages per second) and $0.05 per million for messages above that.
Oh man, Pub/Sub, there's a term I haven't heard in a few years. It was a pretty big thing in XMPP [1] not so long ago (standardized only 2010?), and I remember people being disappointed that the HTTP-based and Google-driven PubSubHubbub [2] was gaining popularity faster. In retrospect, this was a sign of XMPP's loss of steam as a general web protocol :(
Google Cloud Pub/Sub is HTTP(S), so I guess it's based on PubSubHubbub?
From the FAQ [1]: "While Googlers were closely involved in originating PubSubHubbub, its strengths in RSS and content syndication generally are not use cases that Cloud Pub/Sub is designed to address. Aside from the name, they have very little in common."
Yes, you are right. BTW, the use cases are different, but both are using the same core infrastructure that is widely used within Google, which is proven to be scalable and robust :)
PubSub messaging is definitely a PaaS feature more than an IaaS feature, so it's going to be less portable until we have the problem so well surrounded that we get some sort of open source abstraction to the whole problem and all of it's semantics. So, yes, there is a level of lock-in.
However! I know of a number of PaaS products that provide similar functionality, and with some effort, you can build it in AWS or Azure features, or you can build your own on top of RabbitMQ or Apache projects. The characteristics are going to be different, but it's doable. It might be like a MySQL to Postgres migration, or it might be like a MySQL to Mongo migration, but there _is_ a migration. Using a vendor product with unique advantages as a dependency is a known engineering problem with known risks. Take your dependencies carefully, but it's riskier to take no dependencies and fail to deliver a useful product.
[1] http://ecesena.github.io/oscars2015/