Add preliminary job_runs resource#5603
Open
radakam wants to merge 11 commits into
Open
Conversation
Collaborator
Integration test reportCommit: 985e7ba
26 interesting tests: 14 SKIP, 6 flaky, 6 RECOVERED
Top 25 slowest tests (at least 2 minutes):
|
63df005 to
ff9faea
Compare
a899c09 to
3fb23a7
Compare
Contributor
Approval status: pending
|
| // remote identity lives in RunId. Drift on the embedded request fields is | ||
| // suppressed via ignore_remote_changes in resources.yml. | ||
| type JobRunRemote struct { | ||
| jobs.RunNow |
Contributor
There was a problem hiding this comment.
Maybe it's better to embed jobs.Run here because this is what GetRun returns
Contributor
Author
There was a problem hiding this comment.
DoRead now maps the GetRun result back into RunNow shape. It seems that embedding jobs.Run would make the diff not line up as the remote needs to share field paths with the state (which is RunNow-shaped). Do you think it makes sense?
Introduce the job_runs bundle resource backed by the Jobs RunNow API. This is a preliminary skeleton: it wires up the config type, the direct engine resource, registration, and drift handling. Generated schema, validation output, and tests are intentionally left for follow-ups.
Run the schema generator for the new job_runs resource and add real descriptions for the job_runs map, JobRun.lifecycle, and JobRun.python_named_params so the required-annotations guard test passes.
Regenerate the validation rules so job_runs picks up its required field (job_id) and the performance_target enum. Missing required fields surface as warnings via the existing validate:required mutator.
Deploy a bundle with a job and a job_run referencing it, and assert that exactly one RunNow request is made and the returned run id is stored in state. Restricted to the direct engine since job_runs has no Terraform equivalent.
Verify that changing a job_run's configuration re-triggers the run: after the initial deploy, a config change (job_parameters) causes a second, distinct RunNow request on redeploy via the recreate-on-change path.
Several reflection/enumeration tests iterate or hardcode the full resource set and were not updated when the job_runs resource was added, leaving CI red. Add job_runs to each: the run_as allResourceTypes list, the permissions unsupportedResources list, the target-mode mock bundle, the Terraform lifecycle skip list (direct-only), a bind fixture + GetRun mock, and a direct CRUD fixture that triggers a run against a newly created job. The CRUD test also treats job_runs as a no-op delete, since a triggered run cannot be undeployed and stays readable afterward.
Fully populate the GetRunRequest and JobRunRemote literals in job_run.go to satisfy exhaustruct, and regenerate the schema/refschema/apitypes artifacts so the committed job_runs generated files match their generators.
Add job_runs coverage to the two reflection-driven completeness tests: add "job_runs" to the noURL set in TestBundleResourcePluralNamesResolveInWorkspaceURLs (no stable workspace URL yet) and exercise Resources.JobRuns in the StateToBundle test suite.
Add a notebook_change acceptance test that verifies editing a job's notebook_path updates the job (jobs/reset) without re-triggering the run, confirming the run only recreates when its own RunNow config changes. Also extend the basic and redeploy tests with plan/summary/validate traces to make the deploy lifecycle visible in the recorded output.
Map GetRun fields (job_id, job_parameters, overriding parameters) back into
the RunNow shape in DoRead instead of returning an empty RunNow, so the
framework can compute a real diff for the run config.
Add a job_settings snapshot to the job_run config and state: setting
job_settings to a whole-job reference (e.g. ${resources.jobs.my_job}) lets a
change to the targeted job re-trigger the run, since the snapshot is marked
recreate_on_changes. The notebook_change acceptance test now asserts a
notebook_path edit re-triggers the run via this snapshot.
Regenerate schema/validation artifacts for the new job_settings field.
d6b00f5 to
f7ec335
Compare
Avoid partial struct literals in job_run DoRead so exhaustruct passes, and raise the config.Root field-count upper bound now that job_runs snapshots a full job_settings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add a
job_runsbundle resource (direct engine). Deploying it triggers a job run via the JobsRunNowAPI and stores the run id in state.Includes the resource + direct CRUD, lifecycle config, generated schema/validation,
run_ashandling, acceptance tests, and updates to the resource-enumeration tests.Because a run isn't a normal CRUD resource, a few deliberate choices:
RunNow); the run id is the tracked identity.recreate_on_changes— changes re-trigger a fresh run.RunNowre-triggers the run.GetRunechoes the run's job-level and overriding parameters, so those are mapped back and diffed directly; only the request-only inputsGetRunnever returns (idempotency_token,only,performance_target,queue) plus thejob_settingssnapshot areignore_remote_changes.RunNowhas norun_as, so a differing bundlerun_asis rejected early.Re-triggering on job changes (
job_settings)RunNowonly carries the stablejob_id, so a run has no way to notice that the targeted job's definition changed. To support that, the resource has an optionaljob_settingsfield that snapshots the job. Point it at a whole-job reference and any change to the job re-triggers the run (the snapshot isrecreate_on_changes):The snapshot is a local-only input:
GetRunnever returns it, so it'signore_remote_changesto avoid spurious drift.Closes DECO-27355.
(Waiting for completion + run URL are deferred to a later milestone.)
Why
Today, triggering a job as part of
bundle deployrequires custom scripts outside the bundle. This project adds aresources.job_runssection to direct engine so users can declaratively run jobs as part of deployment — e.g. schema migrations, model training, data prep.Tests
job_runs/basic— one run-now, run id stored, clean destroy.job_runs/redeploy— recreate onjob_parameterschange.job_runs/notebook_change— changing the targeted job'snotebook_pathre-triggers the run via thejob_settingssnapshot.TestAll/job_runsdirect CRUD.RunNow/GetRunnow echo job and overriding parameters so reads round-trip.run_as, permissions, target-mode, Terraform lifecycle, bind-support enumeration, workspace-URL completeness (job_runsadded tonoURL— no stable run URL yet), andStateToBundleenumeration tests.