Github and Google recently announced and released the Github archive to BigQuery, liberating a huge dataset of source code in multiple programming languages, and making it easier to query it and discover some insights.

Github explained that the dataset comprises over 3 terabytes of data, for 2.8 million repositories, 145 million commits over 2 billion file paths! The Google Cloud Platform blog gave some additional pointers to give hints about what’s possible to do with the querying capabilities of BigQuery. Also, you can have a look at the getting started guide with the steps to follow to have fun yourself with the dataset.

As I recently joined Google’s developer advocacy team for Google Cloud Platform, I thought I could have a little bit of fun with combining my passion for Apache Groovy with some cool cloudy stuff from Google! Incidentally, Paolo Di Tommaso tweeted about his own experiments with using Groovy with Apache Spark, and shared his code on Github:

I thought that would be a nice fun first little project to try to use Groovy to run a Spark job on Google Cloud Dataproc! Dataproc manages Hadoop & Spark for you: it’s a service that provides managed Apache Hadoop, Apache Spark, Apache Pig and Apache Hive. You can easily process big datasets at low cost, control those costs by quickly creating managed clusters of any size and turning them off where you’re done. In addition, you can obviously use all the other Google Cloud Platform services and products from Dataproc (ie. store the big datasets in Google Cloud Storage, on HDFS, through BigQuery, etc.)

The cat is out the bag: I’m joining Google on June 6th, as a Developer Advocate for the Google Cloud Platform team!

My Groovy friends will likely remember when I launched Gaelyk, a lightweight toolkit for developing Groovy apps on Google App Engine? Since then, I’ve always been a big fan of the Google Cloud Platform (although it wasn’t called that way then) and followed the latest developments of the whole platform. And wohhh, so many new services and products have seen the light of day since my early experiments with App Engine! So there will be a lot to learn, a lot to do, and thus, a lot to advocate! 

Along with the release of Gaelyk 1.2, I’d like to share two interesting links about running functional tests with Geb, and concurrency / parallelism with GPars on Gaelyk.
Gaelyk functional testing with Geb

In the Groovy ecosystem, we’re all aware of the Spock testing framework. On top of Spock, you can use the Geb browser automation library, to easily create functional tests for your web applications, in a nice, readable and expressive fashion.

Thanks to Marcin Erdmann, the Gradle GAE plugin (used by the Gaelyk template project) has been enhanced with support for running functional tests with Spock and Geb. Marcin (who’s blog is actually a fork of my Bloogaey app) has written a small tutorial on how to get started.

In a recent article, I was complaining about the new Google App Engine pricing policy. Obviously, as I have a few applications deployed on App Engine, and as I’m developing Gaelyk, a lightweight toolkit for this platform, I was worried about being heavily affected by those changes.

In this article, I’d like to do a short summary of my experience so far.

I have close to 10 applications deployed on Google App Engine. Most are just demos that nobody ever accesses. But three of them are quite important to me:

I’m quite disappointed by Google App Engine’s new pricing model.

I was aware of the changes, the upcoming prices and quotas, but I wasn’t expecting my small low-trafic apps to go beyond the free quotas, and force me to have to pay for those small Gaelyk apps!

The big problem is the cost of the “frontend instance hours”. An app running all the time, with low trafic, but enough to keep a frontend instance running all day will cost you 30 bucks a month with this new pricing policy.