For a recruiter like me, my passion for tech recruiting is always tempered by the fact that I was never a software developer. But when it comes to understanding the relevance of new technology, I think I actually have an advantage over those who would use it. That’s because since I’m not a software developer, it’s impossible for me to jump on the bandwagon, drink the Kool-Aid, or become obsessed in any way. My perspective will always be a combination of is this relevant to my current or future clients, and is it a relevant skill that my candidates will want to learn?
With that in mind, my picture of Spark is a combination of presentations, industry opinions and business insights.
Is Spark a Fad?
Without the powers of Captain Hindsight to guide us, there is never a clear answer—only time will tell. That being said, I had the opportunity to hear about Spark from the founders at Databricks at the first Seattle Spark meetup. And I understood what they were saying, so it was obviously a great first introduction!
Spark quite obviously offers new and revamped functionalities that are absolutely useful to an extent—I will explain that later. So now the question really is what functionalities can it offer that are different from what’s already out there? The basic non-software developer answer here is that Spark and Spark Streaming allow you to move away from only using batch data. With Spark, you utilize in-memory computing, instead of writing that data to disk like Hadoop MapReduce does. When you consider how fast you can burn music to a flash drive versus music to a CD, you can imagine how writing large data sets to memory can save a lot of time.
Now on the flip side of that, Spark also lets you write applications in Java, Scala and Python. So not only do you now have more flexibility in your hiring, but if you take Scala, for example, you can potentially write a lot less code. That spells awesome in any book!
With that extremely brief overview of Spark, we come back to the question of its lasting ability. Since being developed at UC Berkeley in 2009, what’s happened? Spark has seen what I think is safe to label as pretty fast adoption. It was voted an Apache incubator project in June 2013 and rose to an Apache top-level project in less than a year (February 2014). Moreover, it now has over 200 contributors from 50-plus organizations, which is only the beginning of how the open source community has supported and is supporting this project. I think that is definitely enough to sit up, take notice and think about what Spark could do for you.
Spark: A Solution, But Not a Solution for Everyone
I have heard it all, and what I hear from a lot people who have worked at startups and large companies alike is "we focused on X, when we need to focus on Y." This could be a startup implementing MongoDB when they have no need for it other than the fact that they want to say they are using it; or it can be a large company missing out on the opportunity to become more efficient simply because no one has properly argued the business value to upper management.
So let’s go back to my previous comment about useful to an extent. Spark is a solution, but it is not going to be a solution for everyone. Take that in for a second because it’s important. I know this seems obvious, but as I explained in the previous paragraph, it isn’t always.
Implementing a new technology is always going to cost a company—whether it is the time it takes to migrate the system, or the cost of hiring an expert or training its current employees—and that cost is always there. But if, after looking into Spark, you realize it could be beneficial, don’t let the cost stop you. It’s not smart business to jump on new technology or update just because it is there (after all, there are companies still using COBOL because it works for them), but you also have to balance not falling behind in a rapidly changing and growing industry. The further behind you get, the more costly it can become later—whether that is because your product didn’t scale enough, your competitor used near real-time analytics, or because your server was so old it died and you lost a bunch of data.
Realistically, there are pros and cons to all technical solutions, and as a business, it is your job to investigate new technologies, network in the industry, and learn from the successes and failures of your peers. And as a member of the tech community, it is likewise your job to investigate new technologies, network in the industry, and learn from the successes and failures of your peers. And, yes, that was repetitious on purpose.
The only way to survive in the tech industry is to follow the trends and decide which ones fit your current and future goals, and which don’t. Only after exploring them can you make an educated decision for your future skill set.
What Do You See?
Do I see a greater demand for Spark among our clients? Yes, absolutely. As a passionate technologist, as a job seeker or as a company looking to train its employees, you may be asking what resources are available. The obvious answer is the community! The open source community is really an amazing resource. In Seattle alone, there is a monthly Spark meetup, Spark hack-a-thons, workshops, and probably a lot of events that I never heard about. They also just had a Spark Summit in San Francisco.
My point is that if you want to know more about Spark there are endless resources out there if you just look. Once you discover them, take your time, think about it, talk about it, play with it…and then decide if it is the right technology for you to dig into or for you to convince your company of its magical powers.