Integrating block-sync in an init-container by kushalShukla-web · Pull Request #779 · prometheus/test-infra

kushalShukla-web · 2024-11-08T16:46:07Z

Changes that are introduced in this pr , followed by Create a CLI for uploading and downloading blocks #760 .
Integrated download-key in an init container that downloads the key.yml file from the Prometheus repository. It uses the key.sh script to fetch the pull request and copy the key.yml file into the volumeMount section:

volumeMounts:
  - name: key
    mountPath: /config

Integrated data-downloaderon both pr and released side, which uses the block-sync binary to download pre-generated block data.
Updated the prometheus-builder image, modifying the builder.sh file to copy the key.yml file into the volumeMount section so that the PR section can also download the pre-generated block data.

bboreham

I'm struggling to find my way around. Some trivial points stand out, but I need better orientation.

removed from block-sync:

Please explain each PR in terms of what it does, not what some other PR didn't do.

For example, you could describe the intended purpose of key.sh.

bboreham · 2024-11-12T09:18:18Z

prombench/docs/kind.md

+- **Option 2: Skip downloading data**
+
+   If you don’t need to download data, edit the `3b_prometheus-test_deployment.yaml` file:


Could we instead tell people to create an empty Secret?

bboreham · 2024-11-12T09:18:35Z

prombench/manifests/prombench/benchmark/3b_prometheus-test_deployment.yaml

      initContainers:
      - name: prometheus-builder
-        image: docker.io/prominfra/prometheus-builder:master
+        image: kushalshukla/builder


Fix up before merge.

bboreham · 2024-11-12T09:19:08Z

tools/prometheus-builder/build.sh

    exit 1;
 fi
-
+ 


Spurious change

bboreham · 2024-11-12T09:19:16Z

tools/prometheus-builder/Dockerfile

 RUN chmod +x /go/src/github.com/build.sh

-ENTRYPOINT ["/go/src/github.com/build.sh"]
+ENTRYPOINT ["/go/src/github.com/build.sh"]


Spurious change

bboreham · 2024-11-12T09:23:30Z

tools/prometheus-builder/Dockerfile


 COPY ./build.sh /go/src/github.com/build.sh

+COPY ./key.sh /download-key/key.sh


Why is this in such a different place from the other script?

bboreham · 2024-11-12T09:24:24Z

tools/prometheus-builder/build.sh

+# emptyDir. This file will later be used by the data-downloader init container.
+MKDIR="/config"
+if [ -f "$DIR/key.yml" ]; then
+    echo "File exists."


Who is expected to read this message, and what are they expected to do about it?

bboreham · 2024-11-12T09:26:06Z

prombench/docs/kind.md

+2. Before applying benchmarking objects , You have two choices to make:  
+- **Option 1: Download data from object storage**


How would someone coming to this for the first time understand this choice?
Can we give some context why data might be downloaded?

kakkoyun · 2024-11-13T09:52:36Z

tools/prometheus-builder/build.sh

+# Here, MKDIR is specified in the volumeMount section of the prometheus-builder init container, 
+# where it will copy the key.yml file from the Prometheus directory to the volume section of the
+# emptyDir. This file will later be used by the data-downloader init container.
+MKDIR="/config"


I think you need to change the name.

kakkoyun · 2024-11-13T09:59:22Z

tools/prometheus-builder/build.sh


 git checkout pr-branch

+# Here, MKDIR is specified in the volumeMount section of the prometheus-builder init container, 


It would be better to have this in another script.

from block-sync. Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

2. Modified Prometheus deployment yml file 3. changed Prometheus Builder Docker file Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

…nfig file is empty. Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

prombench/docs/eks.md

kakkoyun · 2024-11-20T11:14:28Z

prombench/docs/eks.md

+Here’s how each option works:
+- **Option 1: Download data from object storage**
+
+   To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml```  with the necessary credentials as per your object storage. This secret enables access to the stored data.


Suggested change

To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml``` with the necessary credentials as per your object storage. This secret enables access to the stored data.

To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml` with the necessary credentials as per your object storage. This secret enables access to the stored data.

prombench/docs/eks.md

prombench/docs/gke.md

kakkoyun · 2024-11-20T11:20:33Z

tools/prometheus-builder/build.sh

+    echo "INFO:: key.yml file is Present on $DIR/key.yml directory."
+    cp  "$DIR/key.yml" "$STORAGE/key.yml"
+else
+    echo "INFO:: key.yml File does not exist on $DIR/key.yml directory."


We should add more context or remove this line. e.g. File does not exists so we won't download XXX

kakkoyun · 2024-11-20T11:21:36Z

tools/prometheus-builder/build.sh

+# where it will copy the key.yml file from the Prometheus directory to the volume section of the
+# emptyDir. This file will later be used by the data-downloader init container.
+if [ -f "$DIR/key.yml" ]; then
+    echo "INFO:: key.yml file is Present on $DIR/key.yml directory."


"It exists so we will download the blocks from XXX."

Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

Removed the code for exiting container when secret is not present. Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

kakkoyun

Looking good. I have some comments.

Also if we merge #810, we will have conflicts make sure we have the relevant changes specific to the PRs.

kakkoyun · 2025-01-29T13:03:47Z

tools/prometheus-builder/key.sh

@@ -0,0 +1,35 @@
+#!/bin/bash


Let's be more specific concerning the filename. key.sh is too generic.

kakkoyun · 2025-01-29T13:04:55Z

tools/prometheus-builder/Dockerfile

-RUN chmod +x /go/src/github.com/build.sh
+COPY ./key.sh /go/src/github.com/key.sh
+
+RUN chmod +x /go/src/github.com/*


We should just give executable permission in this directory. This could create security issues. Just give all the known script names.

kakkoyun · 2025-02-20T09:53:32Z

This needs a rebase @kushalShukla-web

kushalShukla-web · 2025-02-20T12:59:35Z

Hi @kakkoyun @bboreham, we have moved the key into the ConfigMap instead of pulling it from Prometheus, so maybe we can close this pull request ? thats why i dont rebase this.

kushalShukla-web changed the title ~~Added yaml file changes and prometheus builder changes that was removed~~ changes that are not included in Block-sync Nov 8, 2024

kushalShukla-web force-pushed the block-yml branch from 94a489b to 85e464f Compare November 8, 2024 17:02

kushalShukla-web changed the title ~~changes that are not included in Block-sync~~ Integrating Block-sync in an Init-Container. Nov 9, 2024

bboreham reviewed Nov 12, 2024

View reviewed changes

kushalShukla-web force-pushed the block-yml branch 6 times, most recently from 65358a1 to b18501c Compare November 13, 2024 07:10

kakkoyun changed the title ~~Integrating Block-sync in an Init-Container.~~ Integrating Block-sync in an init-container Nov 13, 2024

kakkoyun reviewed Nov 13, 2024

View reviewed changes

kakkoyun changed the title ~~Integrating Block-sync in an init-container~~ Integrating block-sync in an init-container Nov 13, 2024

kushalShukla-web force-pushed the block-yml branch 3 times, most recently from 42e7b36 to 1dfe367 Compare November 14, 2024 12:49

kushalShukla-web added 2 commits November 15, 2024 12:39

Added yaml file changes and prometheus builder changes that was removed

e55d9dd

from block-sync. Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

1. Changed Image name

beec55f

2. Modified Prometheus deployment yml file 3. changed Prometheus Builder Docker file Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

kushalShukla-web force-pushed the block-yml branch 4 times, most recently from 8197ba6 to 1dbaaf1 Compare November 15, 2024 07:20

Added empty secretes ,modified newstore function to normal stop if co…

1c39d4f

…nfig file is empty. Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

kushalShukla-web force-pushed the block-yml branch 2 times, most recently from aab03cb to 18b922f Compare November 19, 2024 15:05

Added --storage.tsdb.retention.size to retain data for certain size.

2c1f884

Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

kushalShukla-web force-pushed the block-yml branch from 18b922f to 2c1f884 Compare November 19, 2024 15:08

kakkoyun reviewed Nov 20, 2024

View reviewed changes

prombench/docs/gke.md Show resolved Hide resolved

kakkoyun reviewed Nov 20, 2024

View reviewed changes

kushalShukla-web force-pushed the block-yml branch from 93ba277 to b795fea Compare November 21, 2024 13:29

minior changes

3e834aa

Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

kushalShukla-web force-pushed the block-yml branch 3 times, most recently from b405f52 to 0f98ebd Compare November 21, 2024 13:49

Removed Image Building from config.yml file.

07476cd

Removed the code for exiting container when secret is not present. Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

kushalShukla-web force-pushed the block-yml branch 2 times, most recently from ec570f4 to 905b6b3 Compare November 22, 2024 15:06

Replaced key.yml with object-config.yml file.

23a7a07

Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>

kushalShukla-web force-pushed the block-yml branch from 905b6b3 to 23a7a07 Compare November 22, 2024 20:32

kushalShukla-web mentioned this pull request Dec 14, 2024

Added Bucket-config as a configmap in a Block-sync init container. #810

Closed

kakkoyun requested changes Jan 29, 2025

View reviewed changes

This was referenced Feb 8, 2025

Added Bucket-config as a configmap in a Block-sync init container. #838

Closed

Added Bucket-config as a configmap in a Block-sync init container. #840

Open

kushalShukla-web mentioned this pull request Feb 20, 2025

Enhance Load Generator to Query Pre-Generated Data Blocks #782

Open

		- Option 2: Skip downloading data

		If you don’t need to download data, edit the `3b_prometheus-test_deployment.yaml` file:


		COPY ./build.sh /go/src/github.com/build.sh

		COPY ./key.sh /download-key/key.sh

		2. Before applying benchmarking objects , You have two choices to make:
		- Option 1: Download data from object storage


		git checkout pr-branch

		# Here, MKDIR is specified in the volumeMount section of the prometheus-builder init container,

	To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name ```object-config.yml``` with the necessary credentials as per your object storage. This secret enables access to the stored data.
	To download data from object storage, create a Kubernetes secret with exact named `bucket-secret` and file name `object-config.yml` with the necessary credentials as per your object storage. This secret enables access to the stored data.

Conversation

kushalShukla-web commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bboreham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kakkoyun Nov 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kakkoyun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kakkoyun commented Feb 20, 2025

Uh oh!

kushalShukla-web commented Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kushalShukla-web commented Nov 8, 2024 •

edited

Loading

kakkoyun Nov 20, 2024 •

edited

Loading

kushalShukla-web commented Feb 20, 2025 •

edited

Loading