[WIP] Proof of Concept: Automated DE VM Setup with terraform + ansible

[lorcanrae]

Proof of Concept: Automated DE VM Setup with Terraform and Ansible

The idea is to automate as much of the DE Setup as possible. Ensuring that students are fully setup on Setup Day and not lingering into the second unit (or later).

Students do the program on a GCP Virtual Machine, providing a controllable environment that lends itself to automation.

Terraform is used to provision the VM, Ansible is used to configure most of the VM.

Open to comment from anyone.

Benefits

• Students don't need to create SSH keys locally and transfer them to the VM, using gcloud compute config-ssh to generate the keys and VS Code Remote - SSH config.
• Students don't need to create a Service Account and Key with far reaching permissions.
  • Use gcloud auth application-default login for ADC credentials. Amend challenges for a Service Account and Key creation applying the principle of least privilege.
• Students are copy pasting a large number of commands during the setup. Given the excitement of the first day, not much of the setup process itself is absorbed.
• All students setup should be standardized. Automating large parts of the setup prevents opportunity for a copy-paste error, simplifying troubleshooting further in the bootcamp.

Drawbacks

• More abstraction of the setup away from students. We could put the original setup doc somewhere as an optional challenge in 0101-Setup-Day or the Project module.
• We don't teach ansible but are using it in the setup.
• The section Authenticate gcloud with Docker 🔗 is not in this setup. I feel this could be comfortably moved to the Docker day.
• Students using Windows need to manually add terraform.exe to PATH. Not that bad, but annoying.

Setup Process

The setup process has three main sections:

1. Local Setup:
   • local setup of communication apps (Slack, Zoom)
   • account creation (GCP, Github)
   • Local gcloud install
   • Local terraform install
   • Use terraform to provision VM
2. Setup of VM Part 1 - Non-dependent software:
   • Authenticate gcloud and ADC
   • Use ansible to install non-dependent software
   • Authenticate GitHub CLI
   • Fork/clone LW dotfiles
3. Setup of VM Part 2 - dependent software:
   • Use ansible to install dependent software (requires GH auth) and modify files (.zshrc, .direnvrc), fork data-engineering-challenges, add upstream remote
   • Manual check of setup - mainly for student interaction, if the ansible playbooks run, everything should work. Can shifted to Kitt for a more interactive, explore your VM challenge.
   • Repo overview and Make install.

Testing

Testing and feedback would be appreciated! Some of the local Windows parts may need fine-tuning.

✔️ Tested Linux setup
✔️ Tested Windows setup

Discussion Points

Currently configured the VM with a static external IP. There are students that change computer or user for whatever reason during the bootcamp. I've tested modifying the config generated by gcloud compute config-ssh with a different user and it works fine. With an ephemeral external IP, this has its drawbacks:

• gcloud won't remove the old ssh-config block if it has been modified, and the list of ssh-hosts would keep growing with an ephemeral external IP.
• Running gcloud compute config-ssh and manually editing the user every time would be a pain
• The nominal cost for a static external IP could be worth the troubleshooting overhead on student edge cases

Dropping the disk size to 100 GB. Should be enough, 150 GB is excessive and it's easy to increase disk size.

TODO

• Confirm if the terraform provisioned VM interacts with GCP services in the same way as a manually provisioned machine. I don't see why it wouldn't, but worth checking.
  • Docker - Cloud Run
  • Spark 2 - VPN and firewall
  • Streaming 2 - Beam
• Confirm challenges that require a Service Account and Key.
  • Airflow 1
  • ELT and DBT

[lorcanrae]

FYI @julesvanrie @AOSXX @Yanka13 @Eschults @gmanchon

[krokrob]

Impressive PoC @lorcanrae ! I won't review the code or the guidelines here. Here are my general comments:

• As you mentioned in the Drawbacks section, this new approach will make the setup very abstract and as a wannabe Data Engineer, students would like to be able to understand their setup environment, don't you think? The benefits won with a lighter and a more straightforward setup may not beat the satisfaction of building a VM setup from scratch for our target students
• That being said, we could use this precious content for 2 use-cases:
  1. An additional setup version, for students that are really struggling with the setup (maybe a 🔴 ) OR to be able to recreate the setup faster in case of incident (computer crash, mis-configured VM,....) AND for former alumni who want a fresh start
  2. Let's make it a challenge in the future Terraform unit! It is a great opportunity to apply the pedagogical approach we love at Le Wagon: build it the hard way to understand what is under the hood, then build it faster with a dedicated/specific framework

Happy to discuss

[lorcanrae]

Hi @krokrob! Thank you for your patience, I've been held up on some other projects 🙏

As you mentioned in the Drawbacks section, this new approach will make the setup very abstract and as a wannabe Data Engineer, students would like to be able to understand their setup environment, don't you think? The benefits won with a lighter and a more straightforward setup may not beat the satisfaction of building a VM setup from scratch for our target students

This is a tricky one and I think depends on the technical level of the student. For some students, absolutely, I think they appreciate understanding their dev environment, but I would put them in the minority. For other students, I think there is very little understanding, they are just copy + pasting code blocks. There is the excitement of the day and we don't explicitly go through the creation of a VM in the lecture. With the amount of copy + pasting, their is a higher chance of introducing variance into the setup.
I may be biased because we interact more with students who are struggling on setup day 😅

My primary concern is students that do not finish the setup on Setup Day and it drags into following sessions, putting a strain on teaching resources and impacting students' learning experience.

An additional setup version, for students that are really struggling with the setup (maybe a 🔴 ) OR to be able to recreate the setup faster in case of incident (computer crash, mis-configured VM,....) AND for former alumni who want a fresh start.
Let's make it a challenge in the future Terraform unit! It is a great opportunity to apply the pedagogical approach we love at Le Wagon: build it the hard way to understand what is under the hood, then build it faster with a dedicated/specific framework

I think this could be the way to go. If we're moving the Terraform module to earlier in the program, adding this as an challenge or optional challenge seems like a good fit.

I recommend that we push these to a different branch (like main-automation or similar) so that the resource is available if needed.

WDYT?

[krokrob]

Hi @lorcanrae thanks for your reply, let's keep it in a dedicated branch 👌. How would you plan to propose this alternative automatic setup to students? during the first lecture?

[lorcanrae]

Great question. Not entirely sure.

I think it would be more indirect, batch managers communicating it in the first session, have an additional info challenge on Kitt to go through the components that were installed and a link to the more manual setup. Maybe from updated marketing info (if this was implemented in the future)?

I don't think it's worth changing the Setup Day Lecture.