Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

- **Fixed** An invalid glob in `--filter` no longer shows its error message twice ([#763](https://github.com/voidzero-dev/vite-task/pull/763)).
- **Changed** The detailed summary from `vp run --verbose` and `vp run --last-details` now shows each underlying cause of an error on its own line ([#761](https://github.com/voidzero-dev/vite-task/pull/761)).
- **Added** Remote caching. Configure an endpoint with the workspace's `cache: { remote: { url } }` or `VP_REMOTE_CACHE_URL`, and choose access with `--remote-cache=off|read|read-write` or `VP_REMOTE_CACHE`. The default is `read` with an endpoint and `off` without one. After a local cache miss, `vp run` looks the task up in the remote cache and, on a hit, restores its outputs and caches it locally. The task output and the run summary show which hits came from the remote cache. A failed read is just a cache miss, with the failure as its reason. In `read-write` mode, `vp run` also uploads the results of successful, cacheable tasks after caching them locally. A failed upload doesn't fail the task; the run summary shows a warning instead. Tasks can opt out with `cache: { remote: false }`. Requests use the proxy environment variables or, on macOS and Windows, the system proxy settings ([#727](https://github.com/voidzero-dev/vite-task/pull/727), [#755](https://github.com/voidzero-dev/vite-task/pull/755), [#756](https://github.com/voidzero-dev/vite-task/pull/756), [#757](https://github.com/voidzero-dev/vite-task/pull/757), [#764](https://github.com/voidzero-dev/vite-task/pull/764)).
- **Added** Remote caching. Configure an endpoint with the workspace's `cache: { remote: { url } }` or `VP_REMOTE_CACHE_URL`, and choose access with `--remote-cache=off|read|read-write` or `VP_REMOTE_CACHE`. The default is `read` with an endpoint and `off` without one. After a local cache miss, `vp run` looks the task up in the remote cache and, on a hit, restores its outputs and caches it locally. The task output and the run summary show which hits came from the remote cache. A failed read is just a cache miss, with the failure as its reason. In `read-write` mode, `vp run` also uploads the results of successful, cacheable tasks after caching them locally. A failed upload doesn't fail the task; the run summary shows a warning instead. Ctrl-C, or a failing task, stops remote cache requests right away, and a task still being looked up doesn't start. Tasks can opt out with `cache: { remote: false }`. Requests use the proxy environment variables or, on macOS and Windows, the system proxy settings ([#727](https://github.com/voidzero-dev/vite-task/pull/727), [#755](https://github.com/voidzero-dev/vite-task/pull/755), [#756](https://github.com/voidzero-dev/vite-task/pull/756), [#757](https://github.com/voidzero-dev/vite-task/pull/757), [#764](https://github.com/voidzero-dev/vite-task/pull/764), [#771](https://github.com/voidzero-dev/vite-task/pull/771)).
- **Fixed** On Windows, environment variable names used by `vp run` now match regardless of ASCII letter case. Assignments in task commands override earlier assignments and inherited variables spelled differently, and `FORCE_COLOR`, `VP_RUN_CONCURRENCY_LIMIT`, and variables requested through `@voidzero-dev/vite-task-client` are found under any spelling ([#747](https://github.com/voidzero-dev/vite-task/pull/747)).
- **Changed** A task's cache settings now go inside `cache`, e.g. `cache: { env: ["NODE_ENV"], input: ["src/**"] }`; `cache: true` is the same as `cache: {}`. `env`, `untrackedEnv`, `input`, and `output` are no longer supported at the top level of a task ([#749](https://github.com/voidzero-dev/vite-task/pull/749)).
- **Fixed** Cached tasks on macOS no longer intermittently fail with exit 2 and `oils I/O error (main): No such process` when a fast command finishes before the shell gets scheduled. The bundled shell that runs task commands is updated to Oils 0.38.0, which fixes this race ([#702](https://github.com/voidzero-dev/vite-task/issues/702), [#703](https://github.com/voidzero-dev/vite-task/pull/703)).
Expand Down
22 changes: 16 additions & 6 deletions crates/vt/src/session/cache/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ pub use display::{
use rusqlite::{Connection, OptionalExtension as _};
use serde::{Deserialize, Serialize};
use tokio::sync::Mutex;
use tokio_util::sync::CancellationToken;
use vt_graph::config::ResolvedGlobConfig;
use vt_path::{AbsolutePath, RelativePathBuf};
use vt_plan::{
Expand Down Expand Up @@ -352,14 +353,15 @@ impl ExecutionCache {
/// remote hit is recorded locally, with its output archive downloaded into
/// `cache_dir`, and is never uploaded. If the local cache has an entry for
/// the task, its miss reason is kept. Otherwise the reason comes from the
/// remote cache.
/// remote cache. Remote requests stop when `cancel_token` is cancelled.
#[tracing::instrument(level = "debug", skip_all)]
pub async fn try_hit(
&self,
cache_metadata: &CacheMetadata,
globbed_inputs: &BTreeMap<RelativePathBuf, u64>,
workspace_root: &AbsolutePath,
cache_dir: &AbsolutePath,
cancel_token: &CancellationToken,
) -> anyhow::Result<Result<CacheHit, CacheMiss>> {
let cache_key = CacheEntryKey::from_metadata(cache_metadata);

Expand Down Expand Up @@ -389,6 +391,7 @@ impl ExecutionCache {
globbed_inputs,
workspace_root,
cache_dir,
cancel_token,
)
.await?
{
Expand Down Expand Up @@ -444,6 +447,7 @@ impl ExecutionCache {
/// and the entry is recorded locally. A fallback entry, a failed
/// validation, or a failed read is a miss. An error while validating
/// counts as a failed read, so the remote entry never fails the task.
#[expect(clippy::too_many_arguments, reason = "forwarded from `try_hit`")]
async fn try_hit_remote(
&self,
endpoint: &Arc<str>,
Expand All @@ -452,10 +456,11 @@ impl ExecutionCache {
globbed_inputs: &BTreeMap<RelativePathBuf, u64>,
workspace_root: &AbsolutePath,
cache_dir: &AbsolutePath,
cancel_token: &CancellationToken,
) -> anyhow::Result<Result<CacheEntryValue, CacheMiss>> {
let fetched = self
.remote_clients
.fetch(endpoint, cache_key, &cache_metadata.execution_cache_key)
.fetch(endpoint, cache_key, &cache_metadata.execution_cache_key, cancel_token)
.await;
let validate = |cache_value: &CacheEntryValue| {
cache_value.validate(&cache_metadata.unfiltered_envs, globbed_inputs, workspace_root)
Expand All @@ -468,7 +473,11 @@ impl ExecutionCache {

let output_archive = match blob_id {
Some(blob_id) => {
match self.remote_clients.download_archive(endpoint, &blob_id, cache_dir).await {
match self
.remote_clients
.download_archive(endpoint, &blob_id, cache_dir, cancel_token)
.await
{
Ok(archive_name) => Some(archive_name),
Err(err) => return Ok(Err(err.into_miss())),
}
Expand Down Expand Up @@ -514,14 +523,15 @@ impl ExecutionCache {
/// as [`Self::record`] does.
///
/// In `read-write` remote mode, the entry is then uploaded to the remote
/// cache. Returns `Ok(Err(_))` if the local update succeeded but the
/// upload failed.
/// cache, until `cancel_token` is cancelled. Returns `Ok(Err(_))` if the
/// local update succeeded but the upload failed.
#[tracing::instrument(level = "debug", skip_all)]
pub async fn update(
&self,
cache_metadata: &CacheMetadata,
cache_value: CacheEntryValue,
cache_dir: &AbsolutePath,
cancel_token: &CancellationToken,
) -> anyhow::Result<Result<(), UploadError>> {
let execution_cache_key = &cache_metadata.execution_cache_key;

Expand All @@ -537,7 +547,7 @@ impl ExecutionCache {
};
let upload = self
.remote_clients
.upload(url, &cache_key, execution_cache_key, &cache_value, cache_dir)
.upload(url, &cache_key, execution_cache_key, &cache_value, cache_dir, cancel_token)
.await;
if let Err(err) = &upload {
tracing::debug!(?err, "remote cache upload failed");
Expand Down
Loading
Loading