The Three-Cloud Hackathon Challenge — What We Learned Running a Two-Day All-Remote Hackathon

Josh Davidson

Principal and Co-Founder, Prime TSR

As our team grows, we thought it would be interesting to put together a hackathon to measure the impact of Covid-19 on Chicago using Cloud-native and Big Data tools. We didn’t want this to be just a “we did a hackathon for a few hours” team-bonding experiment. 

Considering we’re a technology solution provider, we wanted to be able to utilize our core skill-sets to create something valuable for the Chicago community.  We thought the best use of our time is to create tools and visualizations that can drive more insights into the impact of Covid-19 on the Chicago business community.

We took this event seriously, and that’s a big reason why we made this a two-day session during the normal workweek. Giving an entire company the opportunity to take off two full days of work is not the easiest thing to do, but it was important to us, so we made it happen.

Here’s how it worked.

We split into three teams. AWS vs. Azure vs. GCP.  It’s kind of like our own little turf war.  We were all given a general prompt: What is the impact of Covid-19 on new business license requests in Chicago?

We had a few rules to make it more fun and interesting: 

Fair distribution of talent

      1. Just because you’re an expert on AWS, doesn’t mean you’ll be on the AWS team. 
      2. Teams were to be made up of professionals who may not have had the opportunity to previously work together.

Only use real public government data sources

      1. The data sources ranged from data.gov, census.gov, and cityofchicago.org. As long as they were public and accessible via an API or downloadable, it was fair game.

Only use the cloud-native tools you’re assigned

    1. This was cruel for experts who were assigned a platform they weren’t familiar with, but a necessary function for the hackathon exercise.

What we've learned

Send a survey as soon as the hackathon is complete.

We wouldn’t have been able to get real insights into what worked and what didn’t without it. We also used the time during the final team presentations to talk about their process as well as sharing feedback on the hackathon.

Here’s what we learned based on the survey and feedback during their presentations.

It’s the best “happy hour” we’ve had since the creation of the company. 

Overwhelmingly, the hackathon was a success, and people enjoyed it. It pushed people outside of their comfort zones and allowed them to work with new teammates. Also, considering that we did this during standard business hours, it was one of the first times that most of us were not working on client projects at the same time.

I met other Prime TSR people!!! I learned a lot about how the dev teams work together. It's been a while since I have sat through this process with a dev team. It was a fantastic refresher.

Switching our experts into technologies they were unfamiliar with was a great exercise.

Getting an opportunity to learn a new cloud platform and work outside of my typical project space was also a very fun and interesting learning experience.

Technology is hard, and we sometimes take it for granted how hard it is to make it work.

You sometimes forget that it's easy to draw an architecture diagram that points out how to do X, Y, and Z, but it's way more complicated to get into the weeds and get these services to talk to each other and configure them properly. It reminded me that none of this is magic, none of this is particularly "easy," and that was a nice refresher.

It’s a crash course in leadership.

We didn’t assign a leader for the project and let the teams figure out how to do it on their own. It was interesting to see how the roles were assigned and who took it upon themselves to “lead” the team. 

It was a great experience to lead a much different type of team than I’m used to. Leading a diverse project team through the organized chaos that comes with facing a multitude of unknowns in a short period of time. Trust my instincts. Lead with confidence. Flexibility and agility are the keys to short-term project success.

It’s a great way to diversify technical skill-sets and generate awareness for non-technical resources.

For some of our team members, it was their first time in a cloud platform, and that was eye-opening for them. For the rest of the team, it reinforced the idea that hands-on experience with new technologies is by far the best way to learn.

Practice practice practice! You can study and take certifications until your brain melts, but without getting your hands into an environment on a regular basis, your time efficiency is hindered.

An all-remote hackathon worked, but in-person would have been much easier

Each team had a Zoom meeting that stayed on for the entire hackathon and that’s how we communicated. Did it work? Yes. Would in-person have been easier and more productive? Absolutely.  However, we made the most out of the pandemic situation that we could.

What we would do differently next time

Create a social event before “the work” starts.

If you have team members that don’t know each other or haven’t worked together before, it’s useful to create a “happy hour” social event before the hackathon begins instead of sending them into the deep end immediately.

I think it's a really good idea for the team to get to know each other before diving into the work. Who likes to talk? Who's likely to sit in the corner and quietly work? Who do we need to proactively include? I think a self-assessment to be completed beforehand on background, skills, personality, behavior in a team setting, etc. might be helpful.

I would have liked to have lunch (virtual or in-person) where I got to meet my teammates, where we could have chatted and talked about our strengths and weaknesses. It was hard to divide up work when I didn't know anybody.

Starting from scratch is difficult, even with the hackathon being two days long. 

Two days seems like a long time, considering most hackathons are overnight or one-day event. However, two days still isn’t that much time to understand, solve, and build a solution for a problem.

If we started with a solution that is a bit more "built," it would enable us to spend less time on blocking and tackling, so we can use other "newer" stuff like big data, ML, etc.

We spent a lot of our first day just trying to figure out what we wanted to have done. Thankfully, the cloud we used allowed for quick and rapid creation of our CI/CD and deployment process, so we didn't need to spend time on that. But we did seem to spend time spinning our wheels early on.

More constraints and guidance on the problem would have been more productive. 

This would have allowed the teams to spend less time upfront trying to understand the problem they were solving and allowed them to get to work quickly.

Everyone should work with the same data set and source and have the same goal/problem being solved with time to brainstorm upfront and possibly research (no building but generate ideas).

More specific requirements. (i.e., each team works the same goal.) I think the requirements for this first hackathon were too general. Building the solution based on more specific requirements, especially with the time constraints, would be an approach I would consider next time.

Create smaller teams.

We should have followed Jeff Bezos’ “two-pizza rule” where every internal team should be small enough that it can be fed with two pizzas. Our large teams made it harder to collaborate and work together.

I think it would also be interesting if teams were a bit smaller, perhaps 3-4 people, to keep things a little less project-managey (it’s a word now) and a little more fluid. The crosstalk with a group that size is much more manageable than with 10 people, resulting in tighter collaboration among the group.

Technical findings

From a technical perspective, it was interesting to see the different approaches each group took to solve their respective business challenge. In many ways, how the problem was solved depending on the capabilities and ease of use of their assigned platform.

Here is some of the feedback we got from members who were new to the specific cloud platform.

Here’s what we uncovered:

AWS

  • Has powerful data ingestion and integration tools
  • Strong ability to design an data engine that could ingest multiple kinds of data from many sources
  • It’s easy to link QuickSight (visualization tool) with AWS data stores

GCP

  • For a developer, the tools and platforms integrated really well into the solution
  • Was able to quickly use advanced analytical techniques on

Azure

  • Great integration of development tools and platforms in Azure DevOps
  • Very robust data ingestion tools available through Azure Data Factory
  • Rich reporting capabilities for visualizations with PowerBI

Business challenge results

The intelligent tools and visualizations we created all helped us answer the following questions:

What has been the impact of Covid-19 on Chicago business license submission/approvals in 2020 compared to previous years?

License submissions dropped months prior to any Covid-19 cases in Chicago. This is potentially a leading indicator for future business confidence for Covid-19 progression. Also, we found that healthcare-related business licenses actually increased from previous months, and this time last year.

Any impact between new licenses and renewals?

The city is still processing licenses while submissions dropped significantly. There was no apparent backlog of applications, and the city has the capacity for future licenses.

Does the city have a backlog or capacity for new submissions?

Renewals were impacted more significantly than newly-issued licenses.

Hackathons are great team-building and skill-set-building events

It was our first time doing a hackathon, but it definitely won’t be the last. If you’re thinking of running a hackathon, I’m hoping some of these tips will help you make it more successful. On to the next one!