reduced scanning

AGENTS.md

@@ -34,7 +34,7 @@ decodes the JWT token for user ID, queries the GraphQL API for workout data, and
   - Weights: #FF7900 (255,121,0)
   - Reps: #00BBF9 (0,187,249)
   - Sets: #F15BB5 (241,91,181)
-- **Performance**: Reuse HTTP client across requests to maintain connection pooling and avoid TCP overhead. Session-level caching of user preferences (`user_wants_kg`); `getSession` is lazy, not on every command. Fetch packs up to 10 `jday` queries per GraphQL request (aliases) and runs 8 requests concurrently (`JDAY_BATCH_SIZE=10`, `FETCH_CONCURRENCY=8`). Bounded `jrange` listing (`get_dates` with both oldest and latest, e.g. `table deadlift 2026`) splits the span into independent 32-week windows and fetches them concurrently (`jrange_windows`). `table` / `heatmap` / filtered `list` load workout bodies via `get_jdays` (cache hits stay local; misses use the same batched path as `fetch`) instead of one GraphQL request per date. Sequential `jday` fetching of 152 workouts took ~19s; batched+concurrent takes ~1.2s for the workout downloads. Network slowness on a warm cache is date listing + auth, not parsing. HTTP client uses a larger idle pool, TCP_NODELAY, and gzip.
+- **Performance**: Reuse HTTP client across requests to maintain connection pooling and avoid TCP overhead. Session-level caching of user preferences (`user_wants_kg`); `getSession` is lazy, not on every command. Fetch packs up to 10 `jday` queries per GraphQL request (aliases) and runs 8 requests concurrently (`JDAY_BATCH_SIZE=10`, `FETCH_CONCURRENCY=8`). Date listing (`get_dates`) defaults to `-s 0`: cache filenames plus a bounded `jrange` from the last cached date through today (`resolve_date_scan` / `DateScan::{Full,CacheOnly,Hybrid}`). `-s -1` restores a full network listing; bounded ranges then use concurrent 32-week windows (`jrange_windows`). `table` / `heatmap` / filtered `list` load workout bodies via `get_jdays` (cache hits stay local; misses use the same batched path as `fetch`) instead of one GraphQL request per date. Sequential `jday` fetching of 152 workouts took ~19s; batched+concurrent takes ~1.2s for the workout downloads. With a warm cache and default `-s 0`, `table deadlift` stays local (sub-1s); the old 40s cost was unbounded sequential `jrange`. HTTP client uses a larger idle pool, TCP_NODELAY, and gzip.
 - **Rust Implementation**: Used reqwest for HTTP with client reuse, serde for JSON, base64 for JWT decoding, ansi_term for colors, atty for TTY detection. Handled GraphQL responses, error checking, and inline color application during text generation.
 
 ## Referenced Links
@@ -83,6 +83,7 @@ You can look in `weightxreps-client/src/data/generated---db-types-and-hooks.tsx`
 - Optional body weight line in workout parsing; workouts without "@ <number> bw" are allowed and set bw to None
 - Robust workout parsing that treats invalid exercise blocks (lone # or #exercise with no valid sets) as comments
 - Data access control options: `--force-authentication` (`-a`), `--no-network` (`-N`), `--no-cache` (`-C`), `--no-cache-write` (`-W`) for flexible offline/online operation modes
+- **`-s/--scan-days`** (default `0`): how many days to network-scan for new workout dates before using the cache date list. Must appear before the subcommand (`wxrust -s 7 table deadlift`); `list`/`show` `-s` is `--summary`. `-s 0` scans since the last cached date (skips the network if that date is today); `-s N` scans `[today-N, today]`; `-s -1` is the old full-history `jrange` walk. `-s 0` with `--no-cache` or an empty cache falls back to full scan. `--no-network` ignores this flag. Does not detect edits to already-cached days (API has no mtime).
 - Unit-aware parsing: Parser uses cached user unit preference (`user_wants_kg`) to correctly interpret weights without explicit units when reading from cache or importing files, preventing 2.2x multiplier errors in offline mode
  - Table command for PR progression: Displays personal records over time with 1RM calculations (Brzycki formula), date/exercise filtering, age-based color gradient (256-color ANSI), projected weights for rep ranges 1-10, deterministic processing in chronological order, and deduplication of same-day same-rep PRs (keeps only the best weight per day per rep count)
  - Heatmap command: Displays calendar heatmap of workout intensity with mutually exclusive metric options (--sets, --reps, --volume, --weight, --onerm; default: onerm), date/exercise filtering, color scheme options (--green for RGB green gradient, defaulting to solarized table-style gradient), symbol gradients for no-color mode, adapted from clinvoice-rs heatmap implementation
@@ -176,6 +177,7 @@ Recent refactoring extracted common code into helper functions to improve mainta
   - `workouts::resolve_user_wants_kg`: Consolidates logic for determining user weight unit preference, checking network token then falling back to cache.
   - `workouts::get_dates_from_ranges`: Unified logic for parsing date ranges and fetching/calculating dates, used by both `list` and `fetch` commands.
   - `workouts::jrange_windows` / `fetch_jrange_windows`: Split a bounded date range into concurrent `jrange` week-windows (max 32 weeks each).
+  - `workouts::latest_cached_date` / `resolve_date_scan`: Newest cache filename and the `-s/--scan-days` policy (`DateScan::{Full,CacheOnly,Hybrid}`).
   - `workouts::format_cached_jday_text` / `read_cached_jday_text`: Cache file text used by `write_cached_jday` and `fetch --diff`.
   - `fetch::format_text_diff`: Unified-style diff; returns `None` when local and server texts are identical.
   - `utils::create_progress_bar`: Standardized progress bar creation using `indicatif`.
@@ -213,6 +215,6 @@ Unlike the C version which shows separate tables per filter, the Rust implementa
 - Support for other set types (WxD, WxT, etc.).
 - Support for tags, time/distance sets.
 - DELETE keyword handling in cache management.
-- Cache invalidation without a remote mtime: refetch recent dates always, or ask upstream for `updatedAt` on `JLog` / `JRangeDayData`.
+- Cache invalidation without a remote mtime: `-s/--scan-days` finds new dates but not edits to already-cached days; ask upstream for `updatedAt` on `JLog` / `JRangeDayData`.
 
 

README.md

@@ -59,9 +59,14 @@ Create a `credentials.txt` file with your WeightXReps account email on the first
 - `-N, --no-network`: Skip network access, use cache only (workouts and dates come from local cache)
 - `-C, --no-cache`: Skip cache lookup, fetch all data from server (writes still happen unless `--no-cache-write` is also used)
 - `-W, --no-cache-write`: Disable cache writes (reads still happen unless `--no-cache` is also used)
+- `-s, --scan-days <DAYS>`: How many days to scan for new workout dates before using the cache (default: `0`). Must appear before the subcommand, like other global flags (`wxrust -s 7 table deadlift`). `list`/`show` `-s` is still `--summary`.
+  - `-s 0`: scan from the last cached workout through today (skips the network if the cache already has today)
+  - `-s 7`: scan the last 7 days (and still use cached dates for older history)
+  - `-s -1`: list dates from the server over the full requested range
+  - `-s 0` with `--no-cache` or an empty cache falls back to a full scan
 - `--color <always|never|auto>`: Control color output (default: auto, based on TTY)
 
-**Note:** `--no-network` and `--no-cache` are mutually exclusive.
+**Note:** `--no-network` and `--no-cache` are mutually exclusive. `--no-network` ignores `--scan-days`.
 
 ### Commands
 
@@ -87,7 +92,8 @@ Create a `credentials.txt` file with your WeightXReps account email on the first
 #### Fetch Workouts
 
 - Fetch and cache workouts for 2025: `wxrust fetch 2025`
-- Fetch all workouts: `wxrust fetch`
+- Fetch new workouts since the last cached date: `wxrust fetch` (default `-s 0`)
+- Fetch the full history: `wxrust -s -1 fetch`
 - Show diff between local and server: `wxrust fetch --diff 2025-10` (only workouts whose cache file would change)
 - Force re-download: `wxrust fetch --force 2025`
 - Import from text export file: `wxrust fetch --file export.txt`
@@ -105,7 +111,7 @@ Display a progression table showing personal records (PRs) over time for specifi
 - Filter by date range: `wxrust table 2025`
 - Combine date and exercise filters: `wxrust table 2025 deadlift`
 
-`table` (and `heatmap` / filtered `list`) list dates with concurrent `jrange` windows, then load workouts through the same batched `jday` path as `fetch`. Cached days stay local; `--no-network` skips the date listing round-trip entirely.
+`table` (and `heatmap` / filtered `list`) list dates from the local cache and scan only recent days for new workouts (`--scan-days`, default 0). `-s -1` lists dates with concurrent `jrange` windows over the full range. Workout bodies load through the same batched `jday` path as `fetch`; cached days stay local. `--no-network` skips the date listing round-trip entirely.
 
 Arguments are automatically classified as dates or exercise filters:
 - **Date formats**: `YYYY`, `YYYY-MM`, `YYYY.MM`, `YYYYMM`, `YYYY-MM-DD`, `YYYY.MM.DD`, `YYYYMMDD`

dependencies.sh

@@ -5,7 +5,7 @@ SUDO=
 [ "$(id -u)" = 0 ] || SUDO=sudo
 
 NEED=( cargo libssl-dev pkg-config )
-WANT=( rust-gdb entr rust-clippy )
+WANT=( rust-gdb entr rust-clippy rustfmt )
 
 set -x
 $SUDO apt update

smoke/000-help/expected.stdout

@@ -16,6 +16,7 @@ Options:
   -N, --no-network            Do not connect to the server; use local cache only
   -C, --no-cache              Do not read workouts from the local cache
   -W, --no-cache-write        Do not write fetched workouts to the local cache
+  -s, --scan-days <DAYS>      Days to scan for new workouts (0=since last cached, -1=full history) [default: 0]
       --color <COLOR>         When to color output: auto, always, never [default: auto]
   -v, --verbose               Enable debug output
   -h, --help                  Print help

src/api.rs

@@ -1,8 +1,8 @@
+use ansi_term::Colour;
 use async_trait::async_trait;
 use serde::de::DeserializeOwned;
-use ansi_term::Colour;
-use tokio::sync::OnceCell;
 use std::sync::atomic::{AtomicU64, Ordering};
+use tokio::sync::OnceCell;
 
 static TRANSFER_BYTES: AtomicU64 = AtomicU64::new(0);
 static TRANSFER_REQUESTS: AtomicU64 = AtomicU64::new(0);
@@ -25,24 +25,40 @@ fn record_transfer(bytes: usize) {
     TRANSFER_REQUESTS.fetch_add(1, Ordering::Relaxed);
 }
 
-use crate::models::{GraphQLRequest, GraphQLResponse, WorkoutRequest, WorkoutResponse, UserBasicInfoData, User};
 use crate::formatters::STDERR_COLOR_ENABLED;
-use crate::workouts::{write_cached_user_wants_kg, read_cached_user_wants_kg};
+use crate::models::{
+    GraphQLRequest, GraphQLResponse, User, UserBasicInfoData, WorkoutRequest, WorkoutResponse,
+};
+use crate::workouts::{read_cached_user_wants_kg, write_cached_user_wants_kg};
 
 #[cfg_attr(tarpaulin, ignore)]
 #[async_trait]
 pub trait ApiClient: Send + Sync {
-    async fn login_request(&self, request: &GraphQLRequest) -> Result<GraphQLResponse<crate::models::LoginData>, Box<dyn std::error::Error>>;
-    async fn graphql_request<T: DeserializeOwned + 'static>(&self, token: &str, query: &str, variables: Option<serde_json::Value>) -> Result<GraphQLResponse<T>, Box<dyn std::error::Error>>;
-    async fn get_user_info(&self, token: &str) -> Result<crate::models::User, Box<dyn std::error::Error>>;
+    async fn login_request(
+        &self,
+        request: &GraphQLRequest,
+    ) -> Result<GraphQLResponse<crate::models::LoginData>, Box<dyn std::error::Error>>;
+    async fn graphql_request<T: DeserializeOwned + 'static>(
+        &self,
+        token: &str,
+        query: &str,
+        variables: Option<serde_json::Value>,
+    ) -> Result<GraphQLResponse<T>, Box<dyn std::error::Error>>;
+    async fn get_user_info(
+        &self,
+        token: &str,
+    ) -> Result<crate::models::User, Box<dyn std::error::Error>>;
     async fn user_wants_kg(&self, token: &str) -> bool;
 }
 
 fn log_verbose_request(query: &str, variables: Option<&serde_json::Value>, verbose: bool) {
     if verbose {
         let mut output = format!("Query:\n{}", query);
         if let Some(vars) = variables {
-            output += &format!("\nVariables: {}", serde_json::to_string_pretty(vars).unwrap_or("Failed".to_string()));
+            output += &format!(
+                "\nVariables: {}",
+                serde_json::to_string_pretty(vars).unwrap_or("Failed".to_string())
+            );
         }
         let colored = if *STDERR_COLOR_ENABLED {
             Colour::Blue.paint(output).to_string()
@@ -77,6 +93,8 @@ pub struct DataAccess<'a, C: ApiClient> {
     pub use_network: bool,
     pub use_cache: bool,
     pub write_cache: bool,
+    /// Days to scan for new workout dates (`0` = since last cached, `-1` = full history).
+    pub scan_days: i32,
 }
 
 #[derive(Clone)]
@@ -103,9 +121,17 @@ impl ReqwestClient {
 #[cfg_attr(tarpaulin, ignore)]
 #[async_trait]
 impl ApiClient for ReqwestClient {
-    async fn login_request(&self, request: &GraphQLRequest) -> Result<GraphQLResponse<crate::models::LoginData>, Box<dyn std::error::Error>> {
-        log_verbose_request(&request.query, Some(&serde_json::to_value(&request.variables).unwrap()), self.verbose);
-        let response = self.client
+    async fn login_request(
+        &self,
+        request: &GraphQLRequest,
+    ) -> Result<GraphQLResponse<crate::models::LoginData>, Box<dyn std::error::Error>> {
+        log_verbose_request(
+            &request.query,
+            Some(&serde_json::to_value(&request.variables).unwrap()),
+            self.verbose,
+        );
+        let response = self
+            .client
             .post("https://weightxreps.net/api/graphql")
             .json(request)
             .send()
@@ -118,14 +144,20 @@ impl ApiClient for ReqwestClient {
         Ok(body)
     }
 
-    async fn graphql_request<T: DeserializeOwned + 'static>(&self, token: &str, query: &str, variables: Option<serde_json::Value>) -> Result<GraphQLResponse<T>, Box<dyn std::error::Error>> {
+    async fn graphql_request<T: DeserializeOwned + 'static>(
+        &self,
+        token: &str,
+        query: &str,
+        variables: Option<serde_json::Value>,
+    ) -> Result<GraphQLResponse<T>, Box<dyn std::error::Error>> {
         log_verbose_request(query, variables.as_ref(), self.verbose);
         let request_body = if let Some(vars) = variables {
             serde_json::json!({ "query": query, "variables": vars })
         } else {
             serde_json::json!({ "query": query })
         };
-        let response = self.client
+        let response = self
+            .client
             .post("https://weightxreps.net/api/graphql")
             .header("Authorization", format!("Bearer {}", token))
             .json(&request_body)
@@ -139,9 +171,14 @@ impl ApiClient for ReqwestClient {
         Ok(body)
     }
 
-    async fn get_user_info(&self, token: &str) -> Result<crate::models::User, Box<dyn std::error::Error>> {
-        let user =         self.user_info.get_or_try_init(|| async {
-            let query = r#"
+    async fn get_user_info(
+        &self,
+        token: &str,
+    ) -> Result<crate::models::User, Box<dyn std::error::Error>> {
+        let user = self
+            .user_info
+            .get_or_try_init(|| async {
+                let query = r#"
             query {
                 getSession {
                     user {
@@ -150,23 +187,28 @@ impl ApiClient for ReqwestClient {
                 }
             }
             "#;
-            let response: GraphQLResponse<UserBasicInfoData> = self.graphql_request(token, query, None).await?;
-            if let Some(errors) = response.errors {
-                return Err::<User, Box<dyn std::error::Error>>(format!("GraphQL errors: {:?}", errors).into());
-            }
-            // Default to kg if not available
-            if let Some(data) = response.data {
-                let mut usekg = 1;
-                if let Some(session) = data.get_session
-                    && let Some(val) = session.user.usekg {
+                let response: GraphQLResponse<UserBasicInfoData> =
+                    self.graphql_request(token, query, None).await?;
+                if let Some(errors) = response.errors {
+                    return Err::<User, Box<dyn std::error::Error>>(
+                        format!("GraphQL errors: {:?}", errors).into(),
+                    );
+                }
+                // Default to kg if not available
+                if let Some(data) = response.data {
+                    let mut usekg = 1;
+                    if let Some(session) = data.get_session
+                        && let Some(val) = session.user.usekg
+                    {
                         write_cached_user_wants_kg(val != 0);
                         usekg = val;
                     }
-                Ok(User { usekg: Some(usekg) })
-            } else {
-                Err("No data in response".into())
-            }
-        }).await?;
+                    Ok(User { usekg: Some(usekg) })
+                } else {
+                    Err("No data in response".into())
+                }
+            })
+            .await?;
         Ok(user.clone())
     }
 
@@ -177,24 +219,36 @@ impl ApiClient for ReqwestClient {
         let user = self.get_user_info(token).await;
         match user {
             Ok(ref u) => return u.usekg.unwrap_or(1) == 1,
-            Err(_) => return false
+            Err(_) => return false,
         }
     }
 }
 
 #[cfg_attr(tarpaulin, ignore)]
-pub async fn login_request<C: ApiClient>(client: &C, request: &GraphQLRequest) -> Result<GraphQLResponse<crate::models::LoginData>, Box<dyn std::error::Error>> {
+pub async fn login_request<C: ApiClient>(
+    client: &C,
+    request: &GraphQLRequest,
+) -> Result<GraphQLResponse<crate::models::LoginData>, Box<dyn std::error::Error>> {
     client.login_request(request).await
 }
 
 #[cfg_attr(tarpaulin, ignore)]
-pub async fn graphql_request<T: DeserializeOwned + 'static, C: ApiClient>(client: &C, token: &str, query: &str, variables: Option<serde_json::Value>) -> Result<GraphQLResponse<T>, Box<dyn std::error::Error>> {
+pub async fn graphql_request<T: DeserializeOwned + 'static, C: ApiClient>(
+    client: &C,
+    token: &str,
+    query: &str,
+    variables: Option<serde_json::Value>,
+) -> Result<GraphQLResponse<T>, Box<dyn std::error::Error>> {
     client.graphql_request(token, query, variables).await
 }
 
 #[cfg_attr(tarpaulin, ignore)]
 #[allow(dead_code)]
-pub async fn workout_request(client: &reqwest::Client, token: &str, request: &WorkoutRequest) -> Result<WorkoutResponse, Box<dyn std::error::Error>> {
+pub async fn workout_request(
+    client: &reqwest::Client,
+    token: &str,
+    request: &WorkoutRequest,
+) -> Result<WorkoutResponse, Box<dyn std::error::Error>> {
     let response = client
         .post("https://weightxreps.net/api/graphql")
         .header("Authorization", format!("Bearer {}", token))