Skip to content

Integration — Mr. Learn (External LMS Sync)

This domain mirrors learner activity from Mr. Learn — the external Graphy LMS that MAS operates at mrlearn.in — into the Mr. Mentor backend's own PostgreSQL so that the admin dashboard, student dashboard and gamification engine can read course progress without hitting Graphy on every request. It owns a dedicated mrlearn Postgres schema, a per-course sync configuration, an audited sync-run history, a cross-course "students" roll-up, a WhatsApp progress-reminder pipeline, and a one-click "promote a Graphy learner to a full MAS account" flow. Three BullMQ workers drive it: a per-config sync worker, a global new-student fast-sync cron, and a reminder worker.

Status: documented from source on this branch.

Note on naming: in code and in the Graphy API the LMS is referred to interchangeably as Mr. Learn, MrLearn and Graphy. The product brand is "Mr. Learn"; "Graphy" is the underlying SaaS. A sync config is keyed by a Graphy course id, which is what the admin UI labels as a "batch / course". The low-level Graphy session and proxy endpoints (/api/graphy/auth/*, /api/graphy/courses, /api/graphy/proxy, …) belong to the separate Graphy LMS proxy controller; this document covers the sync half (/api/graphy/sync/*) plus the auth-credentials entity it depends on.

Overview

Mr. Learn has no embed/SDK, so the backend logs into Graphy with a single service account, holds the session, and pulls data on a schedule. What this domain does:

  • Stores Graphy service-account credentials (MrLearnAuthCredentials) — encrypted password plus the live c_ujwt / microservice token captured at login. Login/logout/status are handled by the Graphy LMS proxy controller; this domain consumes the stored session via getGraphyLmsService().
  • Per-course sync configs (MrLearnSyncConfig) — one row per Graphy course an admin wants mirrored, with its own interval, "include detailed reports" flag, and an optional reminder schedule.
  • Scheduled + manual sync (MrLearnSyncService) — refreshes the course snapshot, pages through enrolled learners, optionally pulls per-learner detailed reports, and writes them to mrlearn.courses, mrlearn.learners and mrlearn.learner_reports. Every run is audited in mrlearn.sync_runs.
  • New-student auto-provision — a global cron fans out a cheap fast sync to every enabled config, looking only for newly-enrolled learners.
  • WhatsApp reminders (MrLearnReminderService) — nudges learners whose progress is below a threshold to keep going, with full per-send logging.
  • Cross-course student roll-up + MAS account creation — aggregates learners by de-prefixed email across all courses, cross-references Mr. Test (EzExam) submissions, and lets an admin one-click create a User (+ Application) from a Graphy learner.
  • Gamification side-effect — when the detailed report shows a module just completed, the service grants module_completed XP and re-evaluates badges (see student-engagement-gamification).

Personas / roles

Persona What they do Access
Admin / Superadmin Configure sync per course, trigger manual syncs, review runs/learners, send reminders, promote learners to MAS users. All /api/graphy/sync/* routes (admin-guarded).
Scheduler (BullMQ) Runs scheduled syncs, the global new-student cron, and recurring reminders. Internal — no HTTP.
Student Indirect consumer — their dashboard renders progress mirrored here; XP/badges are credited from sync. None directly.

Key concepts & entities

Concept Meaning
Graphy course id The unit we sync against (MrLearnSyncConfig.graphyCourseId). The Graphy _id. User-facing "batch / course".
Sync config Persistent per-course settings: interval, enabled, includeDetailedReports, plus the reminder schedule.
Learner One (graphyCourseId, graphyUserId) enrolment with progress, module breakdown and a phone number.
Report Raw per-learner Graphy report JSON, kept verbatim in mrlearn.learner_reports; summary fields are projected onto the learner row.
Sync run Audit row for one execution: counts, duration, status, error.
Fast vs full sync Fast walks the listing newest-first and early-exits after a streak of already-known learners (new-enrolment detection). Full refreshes progress for everyone.
Monotonic merge A learner's progress / completion / time is never regressed by a later sync — MAX is taken against the existing row.
De-prefixed email Graphy stores org-prefixed addresses (sajantonge2113-rohan@gmail.com); stripEmailOrgPrefix reduces them to rohan@gmail.com for matching MAS users.
Reminder A WhatsApp template nudge for low-progress learners, logged in mrlearn.reminder_logs.

TypeORM entities

Entity File Table Notes
MrLearnAuthCredentials src/entities/mrlearn/MrLearnAuthCredentials.ts mrlearn.auth_credentials One service account; AES-256-GCM encrypted password keyed off JWT_SECRET. Holds cUjwt, sessionId, microserviceToken.
MrLearnSyncConfig src/entities/mrlearn/MrLearnSyncConfig.ts mrlearn.sync_configs Per-course config + reminder schedule. Unique on graphyCourseId.
MrLearnCourse src/entities/mrlearn/MrLearnCourse.ts mrlearn.courses Course snapshot; PK is the Graphy _id string.
MrLearnLearner src/entities/mrlearn/MrLearnLearner.ts mrlearn.learners Enrolment + progress; unique (graphyCourseId, graphyUserId). Holds moduleProgress / courseItemProgress JSONB.
MrLearnLearnerReport src/entities/mrlearn/MrLearnLearnerReport.ts mrlearn.learner_reports Raw report JSON; unique (graphyCourseId, graphyUserId).
MrLearnSyncRun src/entities/mrlearn/MrLearnSyncRun.ts mrlearn.sync_runs Per-run audit. Status: running / success / failed / unauthenticated.
MrLearnReminderLog src/entities/mrlearn/MrLearnReminderLog.ts mrlearn.reminder_logs One row per attempted reminder. Result: sent / skipped / failed.
BatchMrLearnCourse src/entities/BatchMrLearnCourse.ts batch_mrlearn_courses (default schema) Maps a MAS Batch + roadmap item to a Graphy course; cross-schema reference by id only.

Supporting services/utilities: - src/services/MrLearnSyncService.ts — the sync engine. - src/services/MrLearnReminderService.ts — the reminder engine. - src/services/GraphyLmsService.ts — Graphy HTTP client + session management (getCourseLearners, getUserCourseReport, getCourses, getStudents). - src/utils/mrLearnUrl.ts — resolves/normalizes Graphy course URLs (canonicalMrLearnCourseUrl, resolveMrLearnCourseUrl, bestMrLearnCourseUrl). - src/utils/emailMatching.ts — stripEmailOrgPrefix, emailVariantsForUserLookup.

Architecture

flowchart TD
    Admin["Admin Panel (/admin/mrlearn)"] -->|"HTTP, admin JWT"| Routes["GraphyLmsRoutes mounted at /api/graphy"]
    Routes --> Ctrl["MrLearnSyncController"]

    Ctrl -->|"schedule / trigger"| Queue["QueueService (BullMQ)"]
    Ctrl -->|"read/write"| DB[("Postgres schema mrlearn")]
    Ctrl -->|"new-student cron flags"| SysCfg[("SystemConfig")]

    Queue --> QSync["mrlearnSyncQueue"]
    Queue --> QRem["mrlearnReminderQueue"]
    Queue --> QNew["mrlearnNewStudentSyncQueue"]

    QSync --> WSync["mrlearnSync.worker"]
    QRem --> WRem["mrlearnReminder.worker"]
    QNew --> WNew["mrlearnNewStudentSync.worker"]

    WSync --> SyncSvc["MrLearnSyncService"]
    WRem --> RemSvc["MrLearnReminderService"]
    WNew -->|"fast sync per config"| Queue

    SyncSvc --> Graphy["GraphyLmsService"]
    RemSvc --> WA["WhatsAppService (Meta Cloud / AiSensy)"]
    Graphy -->|"session creds"| Creds[("mrlearn.auth_credentials")]
    Graphy -->|"HTTPS"| GraphyAPI["Graphy LMS (mrlearn.in)"]

    SyncSvc --> DB
    SyncSvc -->|"grant XP, eval badges"| XP["StudentProgressService + BadgeService"]
    SyncSvc -->|"completion notify"| Notif["NotificationService"]
    RemSvc --> DB

Data model

erDiagram
    MR_LEARN_SYNC_CONFIG ||--o{ MR_LEARN_LEARNER : "owns via syncConfigId"
    MR_LEARN_SYNC_CONFIG ||--o{ MR_LEARN_SYNC_RUN : "audited by"
    MR_LEARN_SYNC_CONFIG ||--o{ MR_LEARN_REMINDER_LOG : "reminders for"
    MR_LEARN_COURSE ||--o{ MR_LEARN_LEARNER : "course of (by graphyCourseId)"
    MR_LEARN_COURSE ||--o{ MR_LEARN_LEARNER_REPORT : "reports for"
    MR_LEARN_COURSE ||--o{ BATCH_MR_LEARN_COURSE : "mapped to batches"
    MR_LEARN_LEARNER ||--o| MR_LEARN_LEARNER_REPORT : "detailed report"
    MR_LEARN_LEARNER ||--o{ MR_LEARN_REMINDER_LOG : "nudged by"

    MR_LEARN_AUTH_CREDENTIALS {
        uuid id PK
        varchar email UK
        text encryptedPassword
        text cUjwt
        text sessionId
        text microserviceToken
        timestamp authenticatedAt
    }

    MR_LEARN_SYNC_CONFIG {
        uuid id PK
        varchar graphyCourseId UK
        varchar courseTitle
        uuid batchId
        int intervalHours
        boolean enabled
        boolean includeDetailedReports
        timestamp lastSyncedAt
        timestamp nextSyncAt
        varchar lastSyncStatus
        int lastSyncLearnerCount
        boolean reminderEnabled
        int reminderIntervalHours
        int reminderSkipAboveProgress
        varchar reminderTemplateName
        varchar lastReminderStatus
    }

    MR_LEARN_COURSE {
        varchar id PK
        varchar title
        varchar publisher
        boolean published
        varchar courseUrl
        int usersCount
        jsonb raw
        timestamp lastSyncedAt
    }

    MR_LEARN_LEARNER {
        uuid id PK
        uuid syncConfigId
        varchar graphyCourseId
        varchar graphyUserId
        varchar name
        varchar email
        varchar mobile
        decimal progressPercent
        int totalTimeSeconds
        boolean isCompleted
        varchar status
        timestamp enrolledAt
        timestamp startedAt
        timestamp lastAccessedAt
        timestamp completedAt
        jsonb moduleProgress
        jsonb courseItemProgress
    }

    MR_LEARN_LEARNER_REPORT {
        uuid id PK
        varchar graphyCourseId
        varchar graphyUserId
        jsonb report
        timestamp lastSyncedAt
    }

    MR_LEARN_SYNC_RUN {
        uuid id PK
        uuid syncConfigId
        varchar graphyCourseId
        varchar status
        varchar triggeredBy
        timestamp startedAt
        timestamp finishedAt
        int learnersFetched
        int learnersUpserted
        int reportsFetched
        int durationMs
        text error
    }

    MR_LEARN_REMINDER_LOG {
        uuid id PK
        uuid syncConfigId
        uuid learnerId
        varchar graphyCourseId
        varchar graphyUserId
        varchar phone
        varchar triggeredBy
        varchar result
        text detail
        varchar templateName
        uuid whatsappMessageId
        uuid triggeredByUserId
    }

    BATCH_MR_LEARN_COURSE {
        uuid id PK
        uuid batchId
        varchar roadmapItemId
        varchar mrLearnCourseId
        int sequence
        varchar courseUrl
    }

Notable status / enum fields

  • MrLearnSyncConfig.lastSyncStatus: idle | running | success | failed | unauthenticated.
  • MrLearnSyncConfig.lastReminderStatus: idle | running | success | partial | failed.
  • MrLearnSyncRun.status: running | success | failed | unauthenticated.
  • MrLearnSyncRun.triggeredBy: scheduled | manual | boot.
  • MrLearnReminderLog.triggeredBy: manual_single | manual_bulk | scheduled.
  • MrLearnReminderLog.result: sent | skipped | failed.

API surface

All routes live in src/routes/graphyLms.routes.ts, mounted at /api/graphy (src/routes/index.ts line 491). The router applies authMiddleware then adminMiddleware to every route, so all of these require an Admin or Superadmin JWT. The table below lists only the sync routes owned by MrLearnSyncController; the Graphy session/proxy routes (/auth/*, /courses, /users/*, /proxy, etc.) belong to GraphyLmsController and are out of scope here.

Method Path Auth/role Purpose
GET /api/graphy/sync/configs Admin List all sync configs (newest first).
POST /api/graphy/sync/configs Admin Create a config for a course; schedules the repeatable job and kicks an immediate first run.
PATCH /api/graphy/sync/configs/:id Admin Update interval / enabled / detailed-reports / reminder settings; re-schedules sync and reminder jobs when those change.
DELETE /api/graphy/sync/configs/:id Admin Delete a config; ?purgeData=true also deletes learners, reports and reminder logs.
POST /api/graphy/sync/configs/:id/run Admin Enqueue a one-shot manual full sync (202).
POST /api/graphy/sync/by-courses Admin Bulk-create configs and enqueue an immediate sync for many courses at once.
GET /api/graphy/sync/configs/:id/learners Admin Paginated cached learners for a config (search by name/email).
GET /api/graphy/sync/configs/:id/runs Admin Recent sync-run audit rows (newest first).
GET /api/graphy/sync/courses Admin List cached mrlearn.courses rows.
POST /api/graphy/sync/configs/:id/reminders/send Admin Queue a reminder run; body { scope: 'all'|'selected', learnerIds? }.
GET /api/graphy/sync/configs/:id/reminders/logs Admin Recent reminder log rows; filter by learnerId.
GET /api/graphy/sync/students Admin Cross-course student roll-up (aggregated by de-prefixed email, with Mr. Test cross-ref + phone resolution).
POST /api/graphy/sync/students/refresh Admin Hard-refresh: queue a sync for every enabled config. ?full=true for full sync (default fast).
POST /api/graphy/sync/students/reminders/send Admin Queue reminders for { items: [{syncConfigId, learnerId}] } across courses.
POST /api/graphy/sync/students/create-mas-account Admin Promote a Graphy learner (by emailKey) into a MAS User (+ Application if batchId). Optional seedDemo.
POST /api/graphy/sync/students/seed-all-missing Admin Bulk-seed demo data for MAS users whose Mr. Learn + Mr. Test data is empty.
GET /api/graphy/sync/batches Admin Lightweight batch list for the "add to batch" picker.
POST /api/graphy/sync/batches/:batchId/run Admin Sync every Mr. Learn course mapped to a batch (creates configs as needed).
GET /api/graphy/sync/new-student-cron Admin Read the global new-student auto-sync schedule.
PATCH /api/graphy/sync/new-student-cron Admin Toggle / reschedule the global new-student auto-sync (persisted in SystemConfig).

User journeys

Journey 1 — Configure a course for sync

An admin enters a Graphy course id and title, optionally maps it to a batch, sets the interval and whether to pull detailed reports. Creating an enabled config both registers the recurring job and fires an immediate first run.

sequenceDiagram
    participant Admin
    participant Ctrl as MrLearnSyncController
    participant DB as Postgres mrlearn
    participant Q as QueueService
    participant W as mrlearnSync.worker

    Admin->>Ctrl: POST /sync/configs with graphyCourseId and courseTitle
    Ctrl->>DB: findOne config by graphyCourseId
    alt config already exists
        Ctrl-->>Admin: 409 conflict with existing config
    else new
        Ctrl->>Ctrl: clamp intervalHours to 1..720
        Ctrl->>DB: save config status idle
        alt enabled
            Ctrl->>Q: scheduleMrLearnSync configId every intervalHours
            Ctrl->>Q: triggerMrLearnSyncNow configId
            Q->>W: enqueue manual sync job
        end
        Ctrl-->>Admin: 201 created with config
    end

Journey 2 — Scheduled full sync (learners + reports imported, run audited)

The repeatable job ticks. The worker loads the config, then MrLearnSyncService.runSyncForConfig writes a running run row, refreshes the course snapshot, pages learners, optionally pulls per-learner detailed reports, and finalizes the run + config. Each learner upsert uses a monotonic merge; a fresh completion fires a notification and credits XP.

sequenceDiagram
    participant Q as mrlearnSyncQueue
    participant W as mrlearnSync.worker
    participant S as MrLearnSyncService
    participant G as GraphyLmsService
    participant API as Graphy LMS
    participant DB as Postgres mrlearn
    participant XP as Progress and Badges

    Q->>W: scheduled tick with configId
    W->>DB: load config
    alt config disabled and scheduled
        W-->>Q: no-op skipped
    else proceed
        W->>S: runSyncForConfig configId scheduled
        S->>DB: insert sync_run status running
        S->>DB: update config lastSyncStatus running
        S->>G: getCourses then match course id
        G->>API: list courses
        API-->>G: course list
        S->>DB: upsert course snapshot, keep old URL if missing
        loop page through learners until caught up
            S->>G: getCourseLearners course start size
            G->>API: learners page
            API-->>G: learners rows
            S->>DB: upsert each learner with monotonic MAX merge
            Note over S: jitter sleep between pages to dodge rate limits
        end
        S->>DB: backfill mobile from users then from Graphy user list
        opt includeDetailedReports
            loop each learner
                S->>G: getUserCourseReport userId courseId
                G->>API: report
                API-->>G: report JSON
                S->>DB: upsert report and project module progress onto learner
                S->>XP: grant module_completed XP and eval badges for newly done modules
            end
        end
        S->>DB: update sync_run success with counts and duration
        S->>DB: update config success, lastSyncedAt, nextSyncAt, learnerCount
        S-->>W: return run
        W-->>Q: completed
    end

On a fresh completion transition (existing completedAt was null, incoming is set) the upsert resolves the learner email to a MAS User and calls NotificationService.fromTemplate with mrlearn_module_completed. XP grants use skipStreakUpdate: true — a dump can credit XP and badges but never the daily streak.

Journey 3 — New-student auto-provision (global fast-sync cron)

The global cron does no work itself: it fans out a fast sync to every enabled config. Fast mode walks the listing newest-first and breaks once it sees a streak of already-known learners, so it cheaply detects new enrolments and only fetches detailed reports for the newly-added learners.

sequenceDiagram
    participant Cron as mrlearnNewStudentSyncQueue
    participant WN as mrlearnNewStudentSync.worker
    participant DB as Postgres mrlearn
    participant Q as QueueService
    participant S as MrLearnSyncService

    Cron->>WN: scheduled tick
    WN->>DB: find all enabled configs
    alt none enabled
        WN-->>Cron: queued 0
    else
        loop each enabled config
            WN->>Q: triggerMrLearnSyncNow configId fast true
        end
        WN-->>Cron: queued N configs
    end
    Note over S: each fast sync early-exits after 5 consecutive known learners
    Q->>S: fast sync runs, reports limited to newly-added learners

The cron is toggled and re-scheduled via PATCH /sync/new-student-cron, persisted to SystemConfig keys MRLEARN_NEW_STUDENT_SYNC_ENABLED and MRLEARN_NEW_STUDENT_SYNC_INTERVAL_HOURS, and re-registered on boot from those keys.

Journey 4 — Reminder worker nudges inactive learners

For a config with reminderEnabled, a recurring job sends a WhatsApp template to every learner below reminderSkipAboveProgress. Each send resolves a phone (learner row, else MAS user lookup), checks the student's WhatsApp opt-in, builds the 5 body params + URL button, sends, and logs the result.

sequenceDiagram
    participant Q as mrlearnReminderQueue
    participant WR as mrlearnReminder.worker
    participant R as MrLearnReminderService
    participant DB as Postgres mrlearn
    participant Pref as NotificationPreferenceService
    participant WA as WhatsAppService

    Q->>WR: scheduled tick or manual bulk
    WR->>R: runForAll configId
    R->>DB: load config and set lastReminderStatus running
    R->>DB: select learners below skipAboveProgress
    loop each learner
        R->>R: resolveLearnerPhone learner row then users table
        alt no phone
            R->>DB: log result skipped no phone
        else has phone
            R->>Pref: allowsWhatsAppByPhone phone
            alt opted out
                R->>DB: log result skipped opted out
            else allowed
                R->>WA: sendMessage template with body params and URL button
                alt send ok
                    WA-->>R: message id
                    R->>DB: log result sent
                else send error
                    WA-->>R: error
                    R->>DB: log result failed with detail
                end
            end
        end
    end
    R->>DB: update config lastReminder counts, status, nextReminderAt
    R-->>WR: total sent skipped failed

Manual reminders follow the same sendOne path. scope: 'selected' (per-row or multi-select from the students page) routes through runForLearnerIds; scope: 'all' honours the skip-above-progress filter via runForAll. The worker maps scope to triggeredBy = manual_bulk / manual_single so logs distinguish how a send was triggered.

Journey 5 — Map a batch to Mr. Learn courses and sync them

An admin attaches Graphy courses to a MAS batch (rows in batch_mrlearn_courses), then triggers a one-shot sync of every mapped course. Courses without a config get one auto-created (24h, detailed reports on, tagged with the batch); existing configs without a batchId are back-filled.

sequenceDiagram
    participant Admin
    participant Ctrl as MrLearnSyncController
    participant DB as Postgres
    participant Q as QueueService

    Admin->>Ctrl: POST /sync/batches/:batchId/run
    Ctrl->>DB: load batch
    alt batch missing
        Ctrl-->>Admin: 404 batch not found
    else
        Ctrl->>DB: load batch_mrlearn_courses rows for batch
        alt no mapped courses
            Ctrl-->>Admin: 200 nothing mapped
        else
            Ctrl->>DB: load cached titles for course ids
            loop each unique course id
                Ctrl->>DB: find config by graphyCourseId
                alt missing
                    Ctrl->>DB: create config tagged with batch
                    Ctrl->>Q: scheduleMrLearnSync config
                else exists without batch
                    Ctrl->>DB: backfill batchId
                end
                Ctrl->>Q: triggerMrLearnSyncNow config
            end
            Ctrl-->>Admin: 202 queued M of N with created count
        end
    end

Journey 6 — Cross-course student view and promote to MAS account

The students roll-up aggregates every learner by de-prefixed email, resolves phone from learner-or-MAS-user, cross-references Mr. Test submissions (by email and last-10-digits phone), and surfaces the lowest-progress course as the best reminder target. The admin can then one-click promote a learner into a full MAS user.

sequenceDiagram
    participant Admin
    participant Ctrl as MrLearnSyncController
    participant DB as Postgres
    participant MT as mrtest submissions

    Admin->>Ctrl: GET /sync/students with search and filters
    Ctrl->>DB: select all learners with email not null
    Ctrl->>Ctrl: group by stripEmailOrgPrefix into student aggregates
    Ctrl->>DB: lookup users by stripped email for phone and hasMasAccount
    Ctrl->>MT: match by email and by last 10 digit phone
    MT-->>Ctrl: roll numbers and matches
    Ctrl-->>Admin: paged students with avgProgress and lowestProgressCourse

    Admin->>Ctrl: POST /sync/students/create-mas-account emailKey batchId
    Ctrl->>DB: gather matching learner rows for name and phone
    alt none matched
        Ctrl-->>Admin: 404 no learners
    else
        Ctrl->>DB: upsert User, hash default or supplied password, mark verified
        opt batchId provided
            Ctrl->>DB: create or update Application with all onboarding flags true
        end
        opt seedDemo
            Ctrl->>DB: seedDemoData for user
        end
        Ctrl-->>Admin: 200 userId, userCreated, applicationCreated
    end

Background jobs & async

Three dedicated BullMQ queues, each with its own worker (started in src/index.ts), all concurrency: 1 to avoid Graphy / WhatsApp rate-limits. Redis connection from REDIS_HOST / REDIS_PORT.

Queue Worker Trigger Job id pattern Schedule
mrlearnSyncQueue mrlearnSync.worker.ts Per-config repeatable + manual one-shots mrlearn-sync-<configId> (repeatable) every: intervalHours (config; 1h..30d clamp)
mrlearnReminderQueue mrlearnReminder.worker.ts Per-config repeatable + manual one-shots mrlearn-reminder-<configId> (repeatable) every: reminderIntervalHours (default 168h = weekly)
mrlearnNewStudentSyncQueue mrlearnNewStudentSync.worker.ts Single global repeatable mrlearn-new-student-sync every: MRLEARN_NEW_STUDENT_SYNC_INTERVAL_HOURS (default 6h) when enabled 
flowchart LR
    subgraph SCHED["Three worker schedules"]
        A["mrlearnSyncQueue<br/>per config every intervalHours<br/>full sync"]
        B["mrlearnReminderQueue<br/>per config every reminderIntervalHours<br/>default weekly"]
        C["mrlearnNewStudentSyncQueue<br/>one global cron default 6h<br/>fans out fast syncs"]
    end
    C -.->|"triggerMrLearnSyncNow fast"| A

Boot re-registration (src/index.ts): on startup the server re-schedules sync jobs for every enabled config, reminder jobs for every reminderEnabled config, and the global new-student cron based on the SystemConfig flags. Job retention: removeOnComplete/removeOnFail 20 for sync, 50 for reminders.

There are no Socket.IO events or inbound webhooks in this domain.

External integrations

System Used via Env / config Failure behavior
Graphy LMS (mrlearn.in) GraphyLmsService (getCourses, getCourseLearners, getUserCourseReport, getStudents) GRAPHY_BASE_URL (default https://www.mrlearn.in). Session creds in mrlearn.auth_credentials. A 401/403/"microservice token"/"authentication failed"/"no c_ujwt token" / "Email and password required" is detected by isGraphyAuthError. On auth failure the run is marked unauthenticated, the config is auto-disabled (enabled=false) and the repeatable sync job is unscheduled, so we stop burning API calls. Admin must re-login at /admin/mrlearn and re-enable.
Credential encryption aes-256-gcm in GraphyLmsService Key derived from JWT_SECRET. Rotating JWT_SECRET makes the stored password undecryptable → admin must re-login.
WhatsApp WhatsAppService (Meta Cloud API, AiSensy fallback) Template mrlearn_course_progress_reminder (configurable per config), lang default en. Send errors are caught and logged as failed per learner; the run continues.
Notification preferences NotificationPreferenceService.allowsWhatsAppByPhone — Opted-out learners are logged skipped.
Mr. Test (EzExam) Raw SQL against mrtest.submissions Shared Postgres Cross-reference is wrapped in try/catch; failure logs a warning and the student list still returns. See integration-mr-test.
In-app notifications NotificationService.fromTemplate (mrlearn_module_completed) — Non-blocking; failure is logged and the sync continues.
XP / Badges StudentProgressService.grantXp, BadgeService.evaluateForUser — Non-blocking; deduped by the (userId, type, sourceId) unique key in xp_events.

Feature flag: the global new-student auto-sync is gated by SystemConfig.MRLEARN_NEW_STUDENT_SYNC_ENABLED (default off). Per-config sync and reminders are gated by enabled / reminderEnabled columns.

Status lifecycles

Sync run / config sync status

stateDiagram-v2
    [*] --> idle : config created
    idle --> running : sync starts
    running --> success : learners and reports imported
    running --> failed : non-auth error
    running --> unauthenticated : Graphy auth error
    failed --> running : next scheduled or manual run
    success --> running : next scheduled or manual run
    unauthenticated --> idle : admin re-logs in and re-enables
    note right of unauthenticated
        config auto-disabled and
        repeatable job unscheduled
    end note

Reminder config status

stateDiagram-v2
    [*] --> idle
    idle --> running : reminder run starts
    running --> success : all sends ok
    running --> partial : some sent some failed
    running --> failed : nothing sent
    success --> running : next run
    partial --> running : next run
    failed --> running : next run

Reminder log result (per learner)

stateDiagram-v2
    [*] --> evaluating
    evaluating --> skipped : no phone or opted out
    evaluating --> failed : no admin attribution or send error
    evaluating --> sent : WhatsApp accepted
    sent --> [*]
    skipped --> [*]
    failed --> [*]

Edge cases, limits & gotchas

  • Admin-only, everywhere. The whole router is guarded by authMiddleware + adminMiddleware; there is no student-facing endpoint in this domain. Students see mirrored data only via the student dashboard, which reads mrlearn.* indirectly.
  • Monotonic merge is load-bearing. Both upsertLearner and applyReportToLearner take MAX of existing vs incoming progress / time and preserve the first completedAt. Graphy reports a lower percent when a learner re-opens a finished item, so without this a re-sync would visibly regress a student's progress and could re-fire completion XP. Completion notification only fires on the true null → set transition.
  • Fast mode is for enrolment detection, not progress. Fast mode early-exits after FAST_MODE_EXISTING_STREAK_THRESHOLD (5) consecutive known learners and only fetches detailed reports for newly-added learners. Do not use it for scheduled syncs — they need to refresh progress for everyone. The threshold of 5 tolerates Graphy's non-strict ordering.
  • Pagination caps & jitter. Learner paging is capped at MAX_PAGES 200 (20k learners) with LEARNER_PAGE_SIZE 100; randomized sleeps (PAGE_DELAY_MS 600–1800ms, REPORT_DELAY_MS 1200–3500ms) space out calls. concurrency: 1 on every worker means only one Graphy-touching job runs at a time.
  • Phone is hard to get. The per-course listing usually omits phone numbers. The service backfills mobile in three passes: from users.phone (matched on de-prefixed email), from Graphy's /t/api/user/get, and from the per-learner detailed report — and runs the first two again after detailed reports. The reminder service has a fourth last-resort lookup in resolveLearnerPhone.
  • Email prefixing. Graphy stores org-prefixed addresses (org-rohan@gmail.com). All MAS-user matching uses stripEmailOrgPrefix / emailVariantsForUserLookup. The cross-course listStudents aggregation is intentionally done in app code, not SQL, because Postgres POSIX regex doesn't reliably do the lookbehind/lookahead JS does — bounded by total enrolments (a few thousand).
  • Course URL preservation. refreshCourseSnapshot never overwrites a known courseUrl with null when Graphy omits it mid-edit, and self-heals null batch_mrlearn_courses.courseUrl snapshots when a non-null URL is found. The canonical student-facing link is /s/courses/<id>/take (canonicalMrLearnCourseUrl), preferred over Graphy's slug-form spayee:courseUrl.
  • Idempotency. Configs are unique on graphyCourseId (POST returns 409 on dup). Repeatable jobs use stable job ids so reschedule replaces rather than stacks. syncBatch / syncByCourses are re-runnable — they just re-queue existing configs. XP grants dedupe on (userId, 'module_completed', '<courseId>:<moduleId>').
  • create-mas-account gotchas. New users get a bcrypt default password Test@1234, are marked verified + profile-complete (stage 3), and — when a batchId is given — an Application with all onboarding flags forced true (so the dashboard shows no MOU/payment/diagnostic banners). The password column has select: false, so a password rotation on an existing user must be a direct UPDATE (TypeORM save would silently skip it). Existing applications are never re-batched.
  • Auth-failure cascade short-circuit. During detailed-report fetching, an auth error re-throws immediately (rather than logging-and-continuing like other errors) because every subsequent learner would fail identically — failing fast saves N API calls and reduces Graphy lockout risk.
  • Single service account. There is one mrlearn.auth_credentials row in practice; the encrypted password is tied to JWT_SECRET. Cross-schema: batch_mrlearn_courses lives in the default schema and references mrlearn.courses by id only — there is no DB-level FK across schemas.
  • (inferred) The frontend admin surface is /admin/mrlearn in mr-mentor-frontend (referenced in code comments and the graphy-lms skill); this doc covers the backend only.